JP7470844B2 - Audio information replacement system and program - Google Patents

Description

The present invention relates to a voice information replacement system and program.

2. Description of the Related Art Conventionally, there exists a technique for after-recording (so-called dubbing) in which dialogue and music are recorded to match the screen of a story such as a movie.
For example, Patent Document 1 describes an example of a device that allows players to experience voice actors purely while enjoying a sense of unity with the story. This device is provided with a function for playing back voices other than those of the characters selected by the player, and a function for displaying only the subtitles corresponding to the lines of the characters selected by the player at the timing when the characters selected by the player speak. This allows players to experience dubbing by simply speaking in time with the subtitles. In addition, since moving images such as animations and dramas are displayed on the display screen, players can enjoy a sense of unity with the story.

JP 2006-346284 A

The above-mentioned device distinguishes between the characters selected by the player and those not selected by the player, and controls the playback of lines and the output of subtitles for each. Therefore, if the player does not utter the lines of the character selected by the player, the lines in that part will be missing.
Dubbing technology is also used in audiobooks, which are recordings of old tales and the like. Audiobooks can be enjoyed in a variety of ways depending on the age of the child. For example, it is heartwarming to see a young child vocalize lines and sounds that interest them in sync with the playback sound, and the recorded child's voice can become a fond memory for both parent and child. Also, when the child gets a little older, the way they purposely change the lines they are saying can be recorded as a record of their growth. In this way, recordings of children's voices can become both family memories and a record of the child's growth.

However, while audiobooks as educational materials have a button for instructing recording of a voice, they do not have a button for canceling the recording operation after recording has started, or a function for automatically deleting the sound recorded before the cancellation when the recording operation is canceled. Therefore, the user must later open the editing screen to find and delete the unnecessary recording. In addition, if the unnecessary recording remains saved, it may be an obstacle to finding the desired recording, which is one of the factors that reduces the efficiency of the entire voice replacement work.
In addition, although audiobooks have a function to notify users of lines that can be replaced, there is no mechanism for users to check the relative positions of replaceable areas in the audiobook or the number of replaced areas. Therefore, it is not easy to find areas where replacement audio has not been recorded. As a result, a mechanism is needed to improve the efficiency of the entire process of replacing audio by users.

The purpose of the present invention is to make it possible to visually check the number of replaced areas.

The invention described in claim 1 is an audio information replacement system comprising an acquisition means for acquiring an editable audio information file in which a user can participate in replacement, a playback means for playing back the original audio contained in the audio information file, and a presentation means for presenting information to assist in the progress of audio replacement for replaceable areas of the original audio, and which displays the total number of areas in which replacement has been completed for the audio information file being played each time sound replacement for the replaceable areas is completed.
The invention described in claim 2 is a program for enabling a computer to realize the functions of: obtaining an editable audio information file in which a user can participate in replacement; playing the original audio contained in the audio information file; and presenting information to assist in the progress of audio replacement for replaceable areas of the original audio, and the program displays the total number of areas in which replacement has been completed for the audio information file being played each time sound replacement for the replaceable areas is completed.

According to the first aspect of the present invention, the number of replaced areas can be visually confirmed.
According to the second aspect of the present invention, the number of replaced areas can be visually confirmed.

FIG. 1 is a diagram illustrating an overview of a network system assumed in an embodiment. FIG. 2 is a diagram illustrating a configuration example of a terminal used in the embodiment. FIG. 2 is a diagram illustrating the functional configuration of a control unit constituting a terminal. 11 is a flow chart illustrating example process operations performed when editing a downloaded audiobook. 13A and 13B are diagrams illustrating examples of screens at time T1 when an audiobook is selected and time T2 when an instruction to play the audiobook is accepted. FIG. 13 is a diagram illustrating an example of a screen at time T3 when an area where replacement or the like is not possible is being played back, and at time T4 when an area where replacement or the like is possible is being played back. FIG. 13 is a diagram illustrating an example of a screen displayed when a replacement instruction is received. FIG. 13 is a diagram illustrating another example of the screen displayed when a replacement instruction is accepted. 13 is a diagram illustrating an example of a screen displayed in an area where sound can be added. FIG. 13 is a diagram illustrating an example of a screen displayed when an instruction to add a sound is received. FIG. 1 is a diagram illustrating the relationship between areas where replacement etc. is possible and areas where a user has actually instructed replacement etc. (A) shows the arrangement of areas where replacement etc. is possible that are predetermined in an audiobook, and (B) shows the area where a user has actually instructed replacement etc. FIG. 13 is a diagram illustrating an example of a screen display immediately after playback of an audiobook including a replaceable area is started. FIG. 13 is a diagram illustrating an example of a screen display when playback of an audiobook that already has a recording in part of a replaceable area is selected. FIG. 13 is a diagram illustrating a state in which the fifth speech bubble from the top is tapped with an index finger. FIG. 13 is a diagram illustrating an example of a screen display immediately after playback of a replaceable area is started. 13A and 13B are diagrams illustrating a state in which a skip button is operated during playback of a replaceable area. 13A and 13B are diagrams illustrating an example of a screen display after a skip button is operated. 13A and 13B are diagrams illustrating a state in which the record button is operated during playback of a replaceable area. FIG. 13 is a diagram illustrating an example of a screen displayed after operating a record button. FIG. 13 is a diagram illustrating a state in which a skip button is operated during recording. FIG. 13 is a diagram illustrating an example of a screen display after a predetermined time has elapsed since the record button was operated. FIG. 13 is a diagram illustrating an example of a screen displayed after a stop button is operated. FIG. 13 is a diagram illustrating an example of a screen that is automatically displayed when the playback of an audiobook has ended and there is at least one area in which the user has recorded. 13 is a diagram illustrating an example of a screen used when a user specifies an area in which to save a created original audiobook. FIG. FIG. 13 is a diagram showing an example of a screen displayed after the creation of an audiobook has been saved.

The following describes an embodiment of the present invention with reference to the drawings.

<Embodiment>
<Overall system configuration>
FIG. 1 is a diagram for explaining an overview of a network system 1 assumed in the embodiment.
The network system 1 shown in FIG. 1 is composed of an audio file management server 20 connected to the Internet 10, and a terminal 30 operated by a user.
The audio file management server 20 in this embodiment is a server that manages files (hereinafter referred to as "audio files") in which audio data of a person reading a book or the like is recorded for distribution. The audio file management server 20 basically comprises a computer. In the case of FIG. 1, there is one audio file management server 20, but multiple devices may work together to function as the audio file management server 20.

In the present embodiment, audio files that are a unit of transaction with a user are collectively referred to as audiobooks. An audiobook may consist of one audio file or multiple audio files.
In this embodiment, audio books in which the story unfolds through the dialogue of the characters are called "sound stories." Sound stories include, for example, old tales and fairy tales.
In addition, the audio book distributed by the audio file management server 20 is accompanied by information indicating areas where the user can freely insert sounds or areas where the original sounds can be replaced.
Here, an audio book is an example of an audio information file. In the present embodiment, a distinction is made between an area where sound can be inserted and an area where sound can be replaced, and the process of inserting sound is a process of replacing a silent area with an area where sound is present. Therefore, an area where sound can be inserted is, in a broad sense, an example of an area where sound can be replaced.

An audiobook, which has defined areas where insertions or replacements are possible, is an example of an editable sound story file. In the following, the sound recorded in the file before the user inserts or replaces the sound is referred to as the original sound.
The audiobook distributor determines in advance which areas are to be insertable or replaceable. Most of these areas are dialogue. However, the areas in which the user can insert or replace sounds do not have to be all of the dialogue appearing in the audiobook, nor do they have to be limited to the dialogue of a specific character. The areas in which the user can insert or replace sounds may also be part of the narration.

The terminal 30 in this embodiment is a smartphone, tablet terminal, audio player, notebook computer, or the like that is primarily operated by a user who downloads an audiobook.
The terminal 30 has a function of accessing the audio file management server 20 via the Internet 10 and downloading the above-mentioned audio book. However, the connection with the audio file management server 20 may be realized via another device. In the case of this embodiment, the terminal 30 is an example of an audio information replacement system.

<Configuration of terminal 30>
FIG. 2 is a diagram showing an example of the configuration of the terminal 30 used in the embodiment.
The terminal 30 in this embodiment has a control unit 301 that controls the operation of the entire device, a non-volatile memory unit 302 that records data, a display unit 303 used to display a user interface screen and the like, an operation reception unit 304 that receives user operations, a speaker 305 that reproduces electrical signals as sound, a microphone 306 that converts sound into electrical signals, and a communication interface (= communication IF) 307.

The control unit 301 in this embodiment has a CPU (= Central Processing Unit) 311, a ROM (= Read Only Memory) 312 in which firmware and the like are recorded, and a RAM (= Random Access Memory) 313 used as a work area.
The control unit 301 functions as a so-called computer. The ROM 312 is a non-volatile rewritable semiconductor memory.
The storage unit 302 is configured with a non-volatile rewritable semiconductor memory, etc. In the storage unit 302, for example, data of audio books, data of sounds recorded by a microphone 306, etc. are stored.

The display unit 303 is configured with, for example, a liquid crystal display or an organic EL display, and displays information that assists the user in performing operations.
The operation reception unit 304 is composed of, for example, a touch sensor arranged on the surface of the display unit 303, and switches and buttons arranged on the housing.
The communication interface 307 is, for example, a wireless device conforming to a wireless LAN (Local Area Network), Bluetooth (registered trademark), or mobile communication standard.
Incidentally, the control unit 301 and each unit are connected via a bus 308 and signal lines (not shown).
Although not shown, the terminal 30 is equipped with a GPS (Global Positioning System) sensor for acquiring location information, a geomagnetic sensor, an acceleration sensor, a camera for capturing moving images and still images, etc. The location information here is also used to record the location where the sound was recorded.

<Functional configuration of terminal 30>
FIG. 3 is a diagram for explaining the functional configuration of a control unit 301 constituting the terminal 30. As shown in FIG.
The functional modules shown in Fig. 3 are realized through the execution of a program by the CPU 311 (see Fig. 2) and are an example of a program executed by the control unit 301.

One of the programs shown in FIG. 3 is an audiobook acquisition module 321 that acquires an audiobook from the audio file management server 20 (see FIG. 1). The audiobook to be acquired is specified by the user via the operation screen of the terminal 30 (see FIG. 1). Note that, in addition to being acquired from the audio file management server 20, audiobooks may also be acquired from the storage unit 302 (see FIG. 2). The audiobook acquisition module 321 is an example of an acquisition means.

One of the programs shown in FIG. 3 is an audiobook playback module 322 that plays audiobooks.
In this embodiment, there are two types of audiobook playback by the audiobook playback module 322: playback of the original audio and playback of an edited audiobook, and buttons corresponding to each playback are displayed on the display unit 303 (see FIG. 2). The audiobook playback module 322 is an example of a playback means.

One of the programs shown in FIG. 3 includes a replaceable area etc. presentation module 323 that presents to the user areas of the audiobook that can be replaced or inserted with sounds collected by the user.
The replaceable area etc. presentation module 323 here distinguishes the display mode of the dialogue etc. displayed on the display unit 303 (see FIG. 2) between replaceable dialogue etc. and non-replaceable dialogue etc.
For example, lines that cannot be replaced are displayed in a standard thickness and size, while lines that can be replaced are displayed in bold. For example, lines that cannot be replaced are displayed in black text, while lines that can be replaced are displayed in red text. For example, lines that cannot be replaced are not marked, while lines that can be replaced are additionally marked with a distinctive mark.
Furthermore, the replaceable area etc. presentation module 323 displays on the display unit 303 a mark or the like indicating that an area where the sound collected by the user can be inserted is available for insertion.
Moreover, the area where replacement or insertion is possible may be indicated by outputting a specific sound determined in advance. For example, the corresponding position may be notified by a buzzer or the like.

One of the programs shown in FIG. 3 is a replacement instruction receiving module 324 that receives a replacement instruction from the user.
In this embodiment, the replacement instruction is given by operating the record button. When the replacement instruction is accepted, the output of the original audio is stopped, and the device is set in a state in which it is possible to record audio or select pre-recorded audio material. The record button is an example of a second button.
Separate buttons may be provided for recording audio on the spot and for selectively selecting audio material, or the operation may be switched by tapping the record button once or twice. When selectively selecting audio material, playback of the audiobook is also stopped until a selection instruction is received. Recording is stopped by operating the stop button. The stop button is an example of a third button. However, it is also possible to have a mechanism that automatically stops recording after the playback time of the original audio of the line to be replaced has ended.

One of the programs shown in FIG. 3 is a sound material insertion receiving module 325 that receives an instruction from the user to insert a sound material.
In the present embodiment, the recording button is also used to instruct the insertion of sound material. When an instruction to insert sound material is received, the device is also in a state in which it is possible to record voice or select pre-recorded sound material.
Regarding the insertion of sound material, there are two cases: recording the voice on the spot, and selectively specifying the sound material. The operation method is the same as for the replacement instruction.
In the following, the replacement and insertion of sound material will be collectively referred to as "replacement, etc."

One of the programs shown in FIG. 3 includes an edited audiobook storage module 326 that stores an edited audiobook (hereinafter referred to as an "edited audiobook") in the storage unit 302 (see FIG. 2).
The edited audiobook is created separately from the audiobook before editing. Therefore, the audiobook before editing is saved in its pre-edited state. In this embodiment, the date and time of saving, etc. are automatically inserted into the file name of the edited audiobook. A plurality of edited audiobooks saved in this way can be kept as a history of a child's growth. However, the file name may be edited afterwards.

One of the programs shown in FIG. 3 is a support information presentation module 327 that supports the user in replacing voice, etc.
The support information presentation module 327 in this embodiment is a function that presents information on the display unit 303 to support the progress of the user's voice replacement, etc., in addition to the operation of the record button and the stop recording button.
In this embodiment, "information to support the progress of voice replacement by the user" is not information for directly instructing replacement, such as a record button, but information that contributes to improving the efficiency of work related to replacement. In other words, it is information that contributes to understanding the progress of the entire work of replacement and promoting the progress. Here, efficiency includes reduction in the number of steps required to achieve the desired effect and reduction in the time required for the work.

In this embodiment, information to assist the user in proceeding with audio replacement, etc., is provided, for example, a button to instruct the user to skip audio replacement for areas that can be replaced (hereinafter referred to as the "skip button"), the total number of replaceable areas and the number of areas that have been replaced, the positions of replaceable areas and replaced areas on the time axis corresponding to the original audio, and a save destination selection screen that is displayed if audio replacement is performed in any place during playback of the original audio.
By presenting this information, the user can perform tasks such as voice replacement more efficiently.
The support information presentation module 327 is an example of a presentation unit, and the skip button is an example of a first button.

<Processing Operation Example 1>
Here, the processing operations realized by the functional modules other than the support information presentation module 327 (see FIG. 3) among the functional modules included in the terminal 30 (see FIG. 1) will be described.

<Audiobook editing>
4 is a flow chart showing an example of the processing operations performed when editing a downloaded audiobook, where "S" denotes a step.
When the control unit 301 (see FIG. 2) detects a specific operation by the user, it accepts the selection of an audiobook (step 1). One example of the specific operation is an instruction to select an audiobook on a screen that lists downloaded audiobooks.
The control unit 301 that has received the selection instruction determines whether or not the selected audiobook has an area that can be replaced (step 2). The presence or absence of an area that can be replaced can be identified from information attached to the audiobook.

If a negative result is obtained in step 2, the control unit 301 accepts an instruction to play the audiobook and starts playing the audiobook (step 3).
When the reproduction of the audiobook starts, the control unit 301 determines whether the reproduction has ended (step 4).
The control unit 301 continues playing the audiobook while a negative result is obtained in step 4. In this case, since there is no area available for replacement, the audiobook is played back as is.
If step 4 returns a positive result, the audiobook playback is terminated.

On the other hand, if a positive result is obtained in step 2, the control unit 301 accepts an instruction to play the audiobook and starts playing the audiobook (step 5).
Thereafter, the control unit 301 judges whether or not the position being played back is an area where replacement or the like is possible (step 6).
As long as a negative result is obtained in step 6, the control unit 301 repeats the determination.
If a positive result is obtained in step 6, the control unit 301 notifies the user that the area is one in which replacement or the like is possible (step 7).

Next, the control unit 301 (see FIG. 2) determines whether or not an instruction for replacement or the like has been received (step 8).
In this embodiment, the control unit 301 determines whether or not operation of the record button has been detected.
If step 8 gives a negative result, the control unit 301 returns to step 6 and continues playing the audiobook.
On the other hand, if a positive result is obtained in step 8, the control unit 301 accepts the sound material (step 9).
Thereafter, the control unit 301 judges whether or not the reproduction is completed (step 10).
During a negative result at step 10, the control unit 301 returns to step 6 and continues playing the audiobook.

Examples of display screens corresponding to steps 1 to 10 are shown in FIGS.
FIG. 5 is a diagram illustrating an example of a screen at time T1 when an audiobook is selected and at time T2 when an instruction to play the audiobook is accepted.
Time T1 corresponds to step 1 and time T2 corresponds to step 5.
On the screen at time T1, "Fairy Tale 1,""Fairy Tale 2," and "Fairy Tale 3" are displayed as examples of selectable audiobooks, with selection buttons displayed adjacent to each. In Fig. 5, the selection button for "Fairy Tale 1" has a check mark. The screen displayed after the decision button 332 is operated in this state is the screen at time T2.

5, the upper part of the screen at time T2 displays a title 333, a time axis 334, and a slider 335 indicating the playback position. Since playback has not started at time T2, the slider 335 is located at the left end, which is the origin. The slider 335 can also be used to move the playback position.
Three types of buttons are arranged at the bottom of the screen at time T2. Button 336 is a button for playing the original audio, etc. Button 337 is a button used to instruct the user to replace collected audio material, etc. In the case of Figure 5, button 337 is a record button. Button 338 is a button used to instruct the playback of an edited audiobook. In the case of Figure 5, button 338 is a repeat playback button.
In FIG. 5, the user is operating the leftmost button 336 .

FIG. 6 is a diagram for explaining an example of a screen at time T3 when an area where replacement etc. is not possible is being played back, and at time T4 when an area where replacement etc. is possible is being played back.
Time T3 corresponds to a negative result in step 6, and time T4 corresponds to step 7.
In the case of FIG. 6, "Fairy Tale 1" is the fairy tale "Little Red Riding Hood."
On the screen at time T3, since some time has already passed since the start of playback, the slider 335 has moved to near the center of the time axis 334. At time T3, Little Red Riding Hood, a character, speaks a line 339, and the corresponding sound is output from the terminal 30. The line 339 here is, "Oh, Grandma, what big hands you have."
In this embodiment, the dialogue 339 at time T3 is registered as a dialogue that cannot be replaced, etc. For this reason, on the display screen of the terminal 30, a character string representing the content of the dialogue 339 is displayed in a standard thickness and size.

Time point T4 is immediately after time point T3. On the screen at time point T4, a line 340 of the wolf, a character, is displayed, and a corresponding voice is output. The line 340 in this case is "So that you can grasp it well."
In this embodiment, the line 340 at time T4 is registered as a line that can be replaced, but unless the user gives an instruction for replacement, the line 340 is output as is. In fact, in FIG. 6, a voice corresponding to the line 340 is output from the terminal 30.
On the display screen of the terminal 30, the dialogue 340 is displayed in a manner that notifies the user that replacement, etc. is possible. In the case of Fig. 6, the dialogue 340 is changed to bold characters, and at the same time, an oval mark 341 is added behind the dialogue 340. The presence of this mark 341, etc., allows the user to know that the dialogue is possible to be replaced, etc.

7 is a diagram illustrating an example of a screen displayed when a replacement instruction is received. In FIG. 7, parts corresponding to those on the screen at time T4 in FIG. 6 are denoted by the same reference numerals.
Here, the user is operating button 337. The user's operation of button 337 causes terminal 30 to enter a state of waiting for the input of sound material. In Fig. 7, it is assumed that a child's voice is recorded with microphone 306 (see Fig. 2) of terminal 30. For this reason, output of the voice corresponding to lines 340 is stopped. The output of the voice from terminal 30 is stopped in order to prevent the voice output from terminal 30 from being recorded together with the voice of the child.
The output of the sound from the terminal 30 is stopped after the button 337 is operated. In other words, until the button 337 is operated, the sound corresponding to the replaceable dialogue 340 is output from the terminal 30 as shown in FIG.
In the example of FIG. 7, a child is uttering the wolf's lines 340, and this voice is recorded on the terminal 30.

8 is a diagram for explaining another example of the screen displayed when a replacement instruction is received. In Fig. 8, parts corresponding to those in Fig. 7 are denoted by the same reference numerals.
The screen shown in FIG. 8 is similar to FIG. 7 in that it shows replacement of dialogue 340 at time T4, but differs from FIG. 7 in that the time point at which the sound material was acquired and the time point at which replacement editing was performed are different.
In the case of FIG. 8, the fact that replacement is possible is indicated by a character string 341A instead of the mark 341 (see FIG. 7).
Also, in the middle lower part of the screen, a list 342 of candidates for sound materials previously recorded using the terminal 30 or the like is displayed. In the case of Fig. 8, there are three candidates. In the list 342, the file name corresponding to each candidate, a selection button, and a playback button used to check the contents are arranged.

The file names of the candidates shown in FIG. 8 include information on the date and time of recording and the cast, making it possible to identify the content of the sound. In order to distinguish between lines of the same cast recorded on the same day, a serial number is also added to the end of the file name. For example, the first candidate is a sound material recorded at 19:06 on March 7, 2019, and it can be seen that it is the first line of the wolf. The serial number may also mean the order of recording on the recording date.
As in the example of Figure 8, when the file name includes information about the cast, the terminal 30 can selectively display sound material related to the cast that corresponds to the dialogue 340 in the area portion that the user has instructed to be replaced.
Furthermore, when the serial numbers represent replaceable area portions, the terminal 30 can selectively display the associated sound material with pinpoint accuracy.

The file names shown in Fig. 8 include information on the cast, but information indicating the location where the sound material was recorded may be included instead of or together with the cast. For example, a home, a park, etc. may be included in the file name. In addition, the file name may include information indicating whose voice it is and the age of the child at the time of recording. However, this information may only be included in the information attached to the file.
In the example of FIG. 8, the selection button for the second candidate is checked.
In this state, when the decision button 344 is operated, the replacement of the area to be edited is completed. When the back button 343 is operated, for example, the screen returns to the screen before editing.

9 is a diagram for explaining an example of a screen displayed in an area where sound can be added. In Fig. 9, parts corresponding to those on the screen at time T4 in Fig. 7 are denoted by the same reference numerals.
The screen in FIG. 9 corresponds to a time point T5 different from any of the above.
In this embodiment, time T5 is an area where sound can be inserted (i.e., sound can be added), and therefore no dialogue or the like is present.
In FIG. 9, the fact that sound can be added is expressed by a description 346 and a distinctive mark 345.
At this time point T5, the user or a child who wants to add a sound operates button 337.

Fig. 10 is a diagram illustrating an example of a screen displayed when an instruction to add a sound is received. In Fig. 10, parts corresponding to those in Fig. 8 are denoted by the same reference numerals.
In the case of FIG. 10, character string 341B indicates that sound can be inserted.
Also, in the middle lower part of the screen, a list 347 of candidates for sound materials previously recorded using the terminal 30 or the like is displayed. In the case of Fig. 10, there are three candidates. In the list 347, the file name corresponding to each candidate, a selection button, and a playback button used to check the contents are arranged.
The file names in Figure 10 include information about the date and time of recording and the location of recording. For example, the first candidate is known to be a sound material recorded in a park at 14:23 on March 7, 2019.
In the example of FIG. 10, the selection button for the first candidate is checked.
In this state, when the enter button 344 is operated, the insertion of the sound material into the area being edited is completed. When the return button 343 is operated, for example, the screen returns to the screen before editing.
9, when the button 337 is operated, the sound of the scene may be recorded and the recorded sound may be inserted into the corresponding area. The audiobook with the sound inserted is recorded in the storage unit 302 (see FIG. 2) as an edited audiobook.

11 is a diagram illustrating the relationship between areas where replacement etc. is possible and areas where the user has actually instructed replacement etc. (A) shows the arrangement of areas where replacement etc. is possible that are predetermined in an audiobook, and (B) shows the area where the user has actually instructed replacement etc.
11, there are 10 replaceable regions, of which the user has instructed to insert sound material in one region and to replace sound material in two regions. In the remaining regions where replacement, etc. is possible, the original audio, etc., remains as is.

It should be noted that the area in which replacement or the like is possible can be determined regardless of the lines of a specific cast. Therefore, a child can selectively replace only the lines that interest him or her. This feature makes it possible for the audiobook in this embodiment to generate a number of edited audiobooks as the child grows up. For example, when a child is small, only animal sounds are replaced, but as the child grows up, edited audiobooks in which longer lines are also replaced can be recorded and left. In other words, it becomes possible to record the process of growth. It is also possible to record the change in interests that accompanies growth as a difference in the area in which replacement or the like is instructed.
Incidentally, in the case of the audio book in this embodiment, the area parts that can be replaced or the like are determined in advance, so that the user or the child can concentrate on the sounds to be collected.

Returning to the explanation of FIG.
When the reproduction of the audiobook has finished (i.e., a positive result is obtained in step 10), the control unit 301 (see FIG. 2) determines whether or not a replacement or the like has occurred during reproduction (step 11).
If a negative result is obtained in step 11, the control unit 301 ends the process as it is, since this is the same as playing an audio book that has no area that can be replaced or the like.

On the other hand, if a positive result is obtained in step 11, the control unit 301 assigns a file name to the edited audiobook that reflects the results of the editing such as replacement, and saves it (step 12). In this embodiment, the file name of the edited audiobook is composed of the original file name and the editing date.
In this embodiment, when the saving of the edited audiobook is completed, the just saved edited audiobook is automatically played once. This playback is for checking the saved contents. When the repeat playback button 338 (see FIG. 5) is operated, the edited audiobook is repeatedly played.

<Processing Operation Example 2>
In the following, a processing operation specific to the support information presentation module 327 (see FIG. 3) among the functional modules included in the terminal 30 (see FIG. 1) will be described.
The following description focuses on a screen displayed on the display unit 303 (see FIG. 2) of the terminal 30 during playback of an audiobook in which a replaceable area is set.
In the first example of the processing operation, "Little Red Riding Hood" is used as an example of a fairy tale, but in the following, "The Giant Turnip" will be used as an example of a fairy tale.

<Display when playback starts>
FIG. 12 is a diagram illustrating an example of a screen display immediately after the start of playback of an audiobook that includes a replaceable area.
Fig. 12 shows a part of an image displayed on the display unit 303. The display on the screen shown in Fig. 12 corresponds to step 5 (see Fig. 4).
12, a display section 400 is provided in the center of the display unit 303, on which the contents of a story corresponding to an audiobook are displayed. A title 401 is displayed on the display section 400. In the present embodiment, the contents of the story are displayed on the display section 400 in the form of still images or moving images. However, the contents of the story may also be displayed in text on the display section 400.
A time axis 410 indicating the playback time of the story is disposed at the bottom of the display section 400. A slider 411 indicating the current playback position is displayed on the time axis 410. In the case of Fig. 12, since playback has just started, the slider 411 is positioned at the left end corresponding to the beginning position.

12, the area where the sound can be replaced is represented by a speech bubble 412 that is expressed by a picture of a microphone. Ten speech bubbles 412 are arranged along the time axis 410. The display of the speech bubbles 412 that are expressed by a picture of a microphone is realized by the support information presentation module 327 (see FIG. 3).
The display of the balloon 412 informs the user that there are ten replaceable regions.
In the present embodiment, the color and shape of speech bubble 412 change depending on whether the area is already recorded. When the color and shape of speech bubble 412 change depending on whether the area is already recorded, the user can immediately know which areas contain lines that have been recorded and which have not. If the number of recorded areas is simply displayed, it is not possible to know the positions of the recorded areas, but if the color and shape of speech bubble 412 change, it becomes possible to check the positions of the recorded areas.

12 indicates that the audio has not yet been replaced. This can also be confirmed by the display of counter 420 located on the left side of display unit 400. Counter 420 is composed of a microphone mark and a count value. The count value is composed of a denominator that indicates the total number of replaceable areas and a numerator that indicates the number of areas where audio has been recorded.
12, the denominator has a value of “10” and the numerator has a value of “0.” The display of this counter 420 is also realized by the support information presentation module 327.

The display of counter 420 allows the user to easily know that there are 10 replaceable areas and that there is zero areas in which audio has been recorded.
The numerator is incremented each time the number of areas in which audio is recorded increases. The increase in the number of areas in which audio is recorded gives the user a sense of accomplishment, which is expected to increase their motivation to continue recording.
Incidentally, the numerator is counted by area, so even if there are multiple recordings that correspond to a certain area, the numerator is counted as "1."

Furthermore, the user can easily predict whether or not playback of the replaceable area is approaching based on the relationship between the position of slider 411 and the position of balloon 412. By being able to know in advance that playback of the replaceable area is approaching, the user can start recording without panicking.
As a result, the number of times that the playback position needs to be returned in order to record audio in an area that can be replaced is reduced, and the workability regarding replacement is improved.

Fig. 13 is a diagram for explaining an example of a screen display when playback of an audiobook in which a recording has already been made in part of the replaceable area is selected. In Fig. 13, the parts corresponding to those in Fig. 12 are denoted by the same reference numerals.
13 also shows the state immediately after the start of playback of an audiobook that includes a replaceable area, except that the color of the display of the area other than the positions indicated by the fifth and seventh speech bubbles 412 from the beginning has been changed, and the display of counter 420 has been changed to "8/10."

By checking the above display, the user can immediately understand that recordings are recorded in the first through fourth, sixth, and eighth through tenth areas from the beginning.
Furthermore, when the user knows that there is an area where the voice to be replaced has not been recorded, it is possible to motivate the user to try to pinpoint and record the voice in that area.
In this regard, the display described in the first example of the processing operation does not allow the user to know whether the area to be replaced has been recorded or not, so the user ends up recording sound in the area that bothers him/her every time he/she plays the audiobook.

On the other hand, in the case of FIG. 13, the areas where no audio has been recorded can be pinpointed, so that audio recording can be started in the relevant portion simply by tapping the relevant speech bubble 412 on the screen.
14 is a diagram illustrating a state in which the fifth speech bubble 412 from the top is tapped with an index finger. In FIG. 14, parts corresponding to those in FIG. 13 are denoted by the same reference numerals.
After the user taps on the speech bubble 412, an image of the scene corresponding to the fifth speech bubble 412 from the top is displayed on the display unit 400, and playback of the audiobook is resumed. This allows the user to efficiently record the desired audio.
In the case of this embodiment, when speech bubble 412 is tapped, the jump destination is a position just before the position corresponding to the tapped speech bubble 412. In this case, since playback does not immediately start from the position of the tapped speech bubble 412, there is enough time to operate the record button. However, it is also possible to jump to the position corresponding to the tapped speech bubble 412.
Although it is possible to jump the playback position by moving the slider, if the balloon 412 is not displayed, it takes a lot of time and effort to search for the desired position.

A double tap may be required to jump to the location corresponding to speech bubble 412. In this case, a single tap may be used to display all or part of the dialogue corresponding to speech bubble 412.
By displaying all or part of the dialogue, it becomes possible to confirm in advance whether or not it is the dialogue you want to record. If you can check the dialogue in advance, you will no longer need to repeatedly jump around the playback position in order to find the dialogue you want to record, and you can efficiently perform the voice replacement work.
Note that the speech bubble 412 tapped to jump the playback position is not limited to an area in which no audio is recorded. In other words, it is possible to tap a speech bubble 412 corresponding to an area in which audio is already recorded. This is because there are cases in which a user desires to record a different audio even in an area in which audio is already recorded.

<Display of replaceable area during playback>
15 is a diagram for explaining an example of a screen display immediately after playback of a replaceable area is started. The screen display shown in FIG. 15 corresponds to step 7 (see FIG. 4).
In the case of Fig. 15, the color tone of the entire screen is changed to indicate that it is a replaceable area. In the processing operation example 1, the background and notation of the corresponding dialogue 340 (see Fig. 7) were changed. In the case of Fig. 15, by changing the color tone of the entire screen, it is made easier for the user to notice that it is a replaceable area. However, as in the processing operation example 1, it is also possible to change the display of only the part of the corresponding dialogue 340 (see Fig. 7).

15 displays a record button 402, a message 403 prompting an operation, and dialogue 430. Also, on the right side of the display unit 400, from top to bottom, a button 441 for closing the screen, a help button 442, and a skip button 443 are arranged.
15, the message 403 says "Press the button," and requests the user to operate the record button 402. When the record button 402 is operated, recording begins.
In the dialogue 430, the second line, "Grow big, grow sweet," is written in a different color than the lines above and below, indicating that the second line is the one to be recorded. Note that the first line is currently being played back, so it is highlighted, and the second and third lines, which will soon be played back, are displayed with a lower brightness than the first line.

In the case of this embodiment, the screen shown in FIG. 15 is displayed every time a replacement is possible, but there are cases where the user does not wish to replace the corresponding dialogue.
Even in this case, recording will not start unless record button 402 is operated, but it is possible that the user may not like the still images or moving images displayed on display unit 400 being hidden by record button 402 or message 403.
In such a case, the skip button 443 is used.
Fig. 16 is a diagram for explaining a state in which the skip button 443 is operated during playback of a replaceable area. In Fig. 16, the same reference numerals are assigned to the corresponding parts in Fig. 15. Fig. 16 illustrates a state in which the user taps the skip button 443 with a finger.
If the operation of the skip button 443 is detected, a negative result is obtained in step 8 (see FIG. 4). Note that, if the skip button 443 is operated, a negative result is output in the processing of step 6 until the playback of the corresponding area is completed.
Fig. 17 is a diagram for explaining a display example of the screen after operating the skip button 443. In the case of Fig. 17, the display screen returns to displaying the original image. Therefore, the user can enjoy the image. Of course, the playback of the original audio continues.

However, the case where you want to stop recording a voice is not limited to before recording started. For example, you may have started recording by operating the recording button 402 (see FIG. 15), but stopped recording because your child didn't like it. This situation will be explained below with reference to FIG. 18 to FIG. 20.
Fig. 18 is a diagram for explaining the manner in which the record button 402 is operated during playback of a replaceable area. In Fig. 18, the same reference numerals are used to indicate parts corresponding to those in Fig. 15. Fig. 18 illustrates the manner in which the user taps the record button 402 with his/her finger.

Fig. 19 is a diagram for explaining an example of a screen displayed after operating the record button 402. In Fig. 19, the parts corresponding to those in Fig. 18 are denoted by the same reference numerals. The screen display shown in Fig. 19 corresponds to step 9 (see Fig. 4).
By operating the record button 402, a microphone icon 404 and a message 403A indicating that recording is in progress are displayed on the display unit 400. The message 403A displays "Sixth note...". The notation in the second line of the dialogue 430 is changed according to the timing of the utterance of the dialogue. In the example of Fig. 19, the notation up to "Big" is changed to indicate the part that should be uttered.

Normally, the user's voice would be recorded in this situation, but as mentioned above, children may dislike speaking.
If the child does not like to speak, the user can operate the recording stop button, and the sound from when the recording button 402 was tapped is saved in the terminal 30. However, the saved sound is the result of an incomplete recording or sound that is not intended to be used. Therefore, the user needs to find and delete the corresponding sound again after the audiobook playback is finished.
However, such an operation is time-consuming and places a burden on the user.

Fig. 20 is a diagram for explaining the manner in which the skip button 443 is operated during recording. In Fig. 20, the same reference numerals are used to indicate parts corresponding to those in Fig. 19. Fig. 20 illustrates the manner in which the user taps the skip button 443 with a finger.
The screen shown in FIG. 20 corresponds to the forced termination of step 9 (see FIG. 4).
When the skip button 443 is operated, recording is stopped and the display switches to the screen shown in Fig. 17. After this, playback of the original audio is resumed. Note that when the skip button 443 is operated, the original audio may be resumed for the entire line, or for the line that has not yet finished playing at the time the skip button 443 is tapped.
In the case of this embodiment, when the skip button 443 is operated, the audio recorded after the record button 402 is operated is automatically erased. This eliminates the need to delete audio that is not used for replacement after the audiobook has finished playing. In this sense, the efficiency of the audio replacement process is improved.
As described above, the skip button 443 functions as a button for canceling or invalidating the execution of a function.

In the case of this embodiment, once the skip button 443 is operated, even if playback of an area that can be replaced is started after that operation, no display urging the user to record as shown in FIG. 15 is displayed.
If you skip recording because your child doesn't like it, there is a high chance that you will not be able to record in the recordable area the next time or later.In fact, if a screen prompting you to record is displayed each time, the task of skipping recording becomes cumbersome.
Even if no display prompting the user to record is displayed, if the user taps on speech bubble 412 of the dialogue that the user wishes to record, the screen prompting the user to record is displayed again.
In addition, the area affected by the operation of the skip button 443 can be limited to the area where the skip operation is detected. It is preferable that the influence of the skip operation can be switched by the settings of the application program.

Incidentally, when a predetermined time has elapsed since the record button 402 was operated, the display on the display unit 400 switches to a screen for stopping recording.
Fig. 21 is a diagram for explaining an example of a screen display after a predetermined time has elapsed since the operation of the record button 402. In Fig. 21, parts corresponding to those in Fig. 19 are denoted by the same reference numerals.
21 displays a stop button 406 and a message 407 indicating an operation required of the user. The message 407 states, "When the recording is over, press the button," indicating an operation to end the recording.

It is also possible to employ a mechanism that automatically ends recording when the playback of the lines to be recorded ends. However, if such a mechanism is employed, the recording state continues even after the user has finished speaking, and there is a possibility that unintended sounds may be recorded as replacement sounds. In such a case, the user must re-record or delete the recorded sounds.
On the other hand, in the case of this embodiment, recording is ended by operating stop button 406, so that the possibility of unintended sounds being recorded can be reduced.

In this embodiment, the recorded audio is played back so as to replace the entire original audio of the corresponding line, regardless of the recording time. In other words, even if the recording time until the user operates the stop button 406 is shorter than the playback time of the original audio of the replaceable line, no part of the original audio will be played back. This eliminates the need to re-record, improving the ease of replacement.

Fig. 22 is a diagram for explaining a display example of a screen displayed after the stop button 406 (see Fig. 21) is operated. In Fig. 22, the same reference numerals are used to denote parts corresponding to those in Fig. 12.
In the case of this embodiment, screen 450 is displayed each time recording in a replaceable area is completed. Messages such as "Good job!", "You got six sounds!", and "Now, try the next six sounds!" are displayed on screen 450. Screen 450 is displayed for the purpose of increasing a child's motivation. When button 451 labeled "Yes" is operated, screen 450 is closed and playback of the original audio is resumed.
Note that the count value of the counter 420 located on the left side of the screen 450 has been incremented by 1. For example, the numerator of the counter 420 in Fig. 21 was "0", but the numerator of the counter 420 in Fig. 22 has been changed to "1".
In the case of the processing operation example 1, even if the recording is successful, it is unclear which of the replaceable areas the recording was successful in, but in the case of this example, the number of areas where the recording was successful can be confirmed as a specific numerical value. In other words, the number of areas that have not been recorded can also be known. This can increase motivation.

<Display when playback ends>
Finally, a display example at the end of audiobook playback will be described.
Fig. 23 is a diagram illustrating an example of a screen that is automatically displayed when the playback of an audiobook has ended and there is at least one area in which the user has recorded audio. In Fig. 23, the parts corresponding to those in Fig. 12 are denoted by the same reference numerals.
The screen in Fig. 23 is based on the assumption that the created audiobook will be saved automatically. For example, if the user is a child, the user may have difficulty operating the screen that asks the user whether or not to save the audiobook every time playback of the audiobook ends.
Therefore, in FIG. 23, the saving itself is automatically executed by the terminal 30, and the fact that the data has been saved is notified on the screen 460.

23, a screen 460 is displayed to inform the user that an original audiobook has been created and saved. Messages such as "Congratulations!" and "You've added the book to your bookshelf!" are displayed on screen 460. When a button 461 with the words "Add to bookshelf" is operated, screen 460 is closed and switched to a screen for managing a storage area called a bookshelf where audiobooks are stored.
In this embodiment, the creation of an original audiobook also includes the case where a part of an already existing audiobook is modified.
In the case of Fig. 23, the numerator of counter 420 is "8", so the user can confirm that the audiobook created has audio recorded for eight of the ten possible replacement areas.

Fig. 24 is a diagram for explaining an example of a screen used when a user specifies an area in which to save a created original audio book. In Fig. 24, parts corresponding to those in Fig. 23 are denoted by the same reference numerals.
It should be noted that screen 470 shown in FIG. 24 may be displayed when button 461 (see FIG. 23) having the notation "To Hondana" is operated.
Although the automatic saving of created audiobooks is a convenient function, it is possible that a user may have difficulty finding a desired audiobook, for example, when audiobooks of different users are mixed together in one storage location.
Therefore, in FIG. 24, a screen 470 is provided for the user to specify a save destination.

24, a message "Choose a book shelf" indicating the operation required of the user and folders named "Book shelf", "Taro", and "Hanako" as candidates for storage destinations are displayed. Screen 470 also has a button 471 labeled "Change" and a button 472 labeled "Add".
In this embodiment, the "Bookshelf" folder is for storing audiobooks purchased by the user, the "Taro" folder is for storing audiobooks whose voice has been replaced with Taro's voice, and the "Hanako" folder is for storing audiobooks whose voice has been replaced with Hanako's voice.
In the example of Fig. 24, the "Taro" folder is selected as the save destination. If it is desired to change the save destination, operating button 471 changes the selected folder in turn. Operating button 472 saves the created audiobook in the selected folder.

FIG. 25 shows an example of a screen displayed after the creation of an audio book has been saved.
In FIG. 25, parts corresponding to those in FIG. 13 are denoted by the same reference numerals.
The screen shown in FIG. 25 is at the point when the playback of the audiobook has ended, so that the slider 411 has reached the right end of the time axis 410 .
In addition, because there are areas where audio has been recorded, the color of the display of the areas other than the positions indicated by the fifth and seventh speech bubbles 412 from the beginning is changed to a color indicating the presence of recording, and at the same time the display of counter 420 is changed to "8/10."
This display allows the user to visually confirm the results of the recording in the audiobook that has just been created.

<Other embodiments>
Although the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the scope described in the above-mentioned embodiments. It is clear from the claims that various modifications and improvements to the above-mentioned embodiments are also included in the technical scope of the present invention.

For example, in the above embodiment, the terminal 30 (see FIG. 1) was described as an example of an audio information replacement system, but it is also possible for the audio file management server 20 (see FIG. 1) to function as an example of an audio information replacement system. In this case, the terminal 30 and the sound collection device 40 function simply as input/output devices. It is also possible to execute some of the above-mentioned modules on the audio file management server 20 and execute the rest on the terminal 30. In this case, the audio file management server 20 and the terminal 30 are an example of an audio information replacement system.

In the embodiment described above, the edited audiobook is recorded in storage unit 302 (see FIG. 2), but it may also be stored on audio file management server 20 (see FIG. 1) or another server (not shown) on the Internet 10 (see FIG. 1).
In the above embodiment, the area that can be replaced is a dialogue portion, but the area that can be replaced is not limited to a dialogue portion.

In the above embodiment, Otomonogatari is given as an example of an audio information file, but the above technology can also be applied to the playback of learning materials in which areas where the original audio can be replaced are set in advance. The learning materials here include language learning materials. The above-mentioned Otomonogatari is also a type of language learning material.
There are no restrictions on the age of the learners who use language learning materials, but using them for language learning for children from preschool to elementary school age is expected to help them acquire the habit of learning in a fun way.
The language learning materials are not limited to those for learning English or other foreign languages, but may also be for learning Japanese. There are various learning purposes, such as learning hiragana, words, proverbs, songs, tongue twisters, famous quotes, etc.
In addition, the format of the teaching materials is not limited to a story format, but may include a format consisting of a section where a model audio is played and a section where the pronunciation imitates the model, or a format in which questions and answers to the questions are pronounced.

・Note 1
An audio information replacement system comprising: an acquisition means for acquiring an editable audio information file in which a user can participate in replacement; a playback means for playing back the original audio contained in the audio information file; and a presentation means for presenting information to assist in the progress of audio replacement for areas of the original audio that can be replaced.

・Appendix 2
2. The audio information replacement system according to claim 1, wherein when an operation of a first button instructing to skip replacement is detected during playback of the area that can be replaced, the display prompting replacement is removed from the screen.

Appendix 3
The audio information replacement system described in Appendix 2, in which even if operation of a second button to instruct recording of sound is detected during playback of the area that can be replaced, when operation of the first button is detected, the recording process is terminated and the sound recorded after the operation of the second button is erased.

Appendix 4
4. The audio information replacement system according to claim 2 or 3, wherein after operation of the first button is detected, the display prompting replacement is not output to the screen even if playback of another area that can be replaced begins.

Appendix 5
2. The audio information replacement system of claim 1, wherein when the operation of a third button to end recording is detected during audio recording, the entire original audio corresponding to the area that can be replaced is replaced with the sound recorded up until the operation of the third button was detected.

Appendix 6
2. The audio information replacement system according to claim 1, wherein each time sound replacement for the area that can be replaced is completed, the total number of areas that have been replaced for the audio information file being played is displayed.

Appendix 7
The audio information replacement system described in Appendix 1, which displays on the screen a message indicating that the audio has been saved as a new file if there is a history of new replacement in any of the areas that can be replaced when audio playback in the audio information file has ended.

Appendix 8
2. The audio information replacement system according to claim 1, wherein the position of the area that can be replaced and whether or not the sound in the area is to be replaced are indicated on a time axis that indicates the playback time of the audio information file.

Appendix 9
9. The audio information replacement system according to claim 8, wherein when one of the indications indicating the areas that can be replaced is selected, playback of the audio information file begins from the corresponding position.

Appendix 10
The audio information replacement system described in Appendix 1, which displays a screen to accept the selection of a destination to save the audio information file if there is a history of new replacement in any of the areas that can be replaced when audio playback of the audio information file has ended.

Appendix 11
A program for enabling a computer to realize the following functions: a function for a user to participate in replacement and obtain an editable audio information file; a function for playing back the original audio contained in the audio information file; and a function for presenting information to assist in the progress of audio replacement for areas of the original audio that can be replaced.

1...Network system, 10...Internet, 20...Audio file management server, 30...Terminal, 321...Audiobook acquisition module, 322...Audiobook playback module, 323...Replaceable area presentation module, 324...Replacement instruction reception module, 325...Audio material insertion reception module, 326...Edited audiobook storage module, 327...Support information presentation module

Claims (2)

An acquisition means for acquiring an editable audio information file in which a user can participate in the replacement;
a playback means for playing back the original audio included in the audio information file;
a presentation means for presenting information to assist in the progress of the voice replacement for a region of the original voice that can be replaced;
Equipped with
The audio information replacement system displays the total number of areas in which replacement has been completed for the audio information file being played back each time sound replacement for the areas that can be replaced is completed. On the computer,
A function for users to participate in the replacement and obtain editable audio information files;
A function of playing back the original audio contained in the audio information file;
providing information to assist in the progression of the audio replacement for areas of the original audio that can be replaced;
This is a program to achieve the above.
The program displays the total number of areas in which replacement has been completed for the audio information file being played back, each time sound replacement for the areas that can be replaced is completed.