JP7828984B2 - Processing apparatus, processing program, and processing method - Google Patents

Description

This disclosure relates to a processing device, a processing program, and a processing method for outputting speech information associated with a selected object.

Video distribution systems via the internet have been known for some time. For example, Patent Document 1 describes a "video distribution system characterized by comprising: a recruitment management unit that notifies a user terminal of recruitment requirements including video distribution conditions and acquires submitted videos from the user terminal; a video analysis unit that analyzes whether the submitted videos are eligible for distribution and designates eligible submitted videos as distribution videos; and a video distribution management unit that distributes the distribution videos."

Japanese Patent Publication No. 2022-180967

Therefore, based on the technologies described above, this disclosure aims to provide a processing device, processing program, and processing method that are more user-friendly for users such as recipients, through various embodiments.

According to one aspect of this disclosure, a processing device is provided comprising at least one processor, wherein the at least one processor is configured to receive speech information from a transmitting terminal device via a communication interface, associated with each of a plurality of objects included in content generated in the transmitting terminal device, to select at least one of the plurality of objects via an input interface, and to perform processing for outputting speech information associated with the selected at least one object when outputting the speech information via an output interface.

According to one aspect of this disclosure, a processing program is provided that causes a computer comprising at least one processor to function as follows: the computer receiving speech information from a sender terminal device via a communication interface, associated with each of a plurality of objects included in content generated at the sender terminal device; selecting at least one of the plurality of objects via an input interface; and, when outputting the speech information via an output interface, to perform processing to output the speech information associated with the selected at least one object.

According to one aspect of this disclosure, a processing method is provided for a computer comprising at least one processor, the method being performed by the at least one processor, the method comprising: receiving speech information input from a sender terminal device via a communication interface, associated with each of a plurality of objects included in content generated in a sender terminal device; selecting at least one object from the plurality of objects via an input interface; and outputting speech information associated with the selected at least one object when outputting the speech information via an output interface.

This disclosure provides a more user-friendly processing device, processing program, and processing method for users such as recipients.

The effects described above are merely illustrative for illustrative purposes and are not limiting. In addition to, or in lieu of, any other effects described herein or those obvious to those skilled in the art may also be achieved.

Figure 1A is a diagram showing an overview of the processing according to the processing system 1 according to the embodiment of this disclosure. Figure 1B is a block diagram showing the configuration of a processing system 1 according to one embodiment of the present disclosure. Figure 2A is a block diagram showing the configuration of a server device 100 according to one embodiment of the present disclosure. Figure 2B is a block diagram showing the configuration of a terminal device 200 according to one embodiment of the present disclosure. Figure 3 is a schematic diagram showing information transmitted from the sender terminal device 200-1 as transmission information according to one embodiment of the present disclosure. Figure 4 is a diagram showing a processing sequence executed by a processing system 1 according to one embodiment of the present disclosure. Figure 5 is a diagram showing the processing flow performed in a receiver terminal device 200-2 according to one embodiment of the present disclosure. Figure 6 shows an example of a screen output in a sender terminal device 200-1 according to one embodiment of the present disclosure. Figure 7 shows an example of a screen output in a receiver terminal device 200-2 according to one embodiment of the present disclosure. Figure 8A is a diagram showing a processing sequence executed by a processing system 1 according to one embodiment of the present disclosure. Figure 8B is a diagram showing a processing sequence executed by a processing system 1 according to one embodiment of the present disclosure. Figure 9 is a diagram showing an overview of the processing according to the processing system 1 according to the embodiment of this disclosure.

1. Overview of Processing System 1 The processing system 1 described herein is used to output speech information associated with an object desired by the receiver for content transmitted from the sender. For example, the processing system 1 is used to output only the audio associated with a character object when the receiver selects one of the character objects appearing in video content transmitted from the sender.

Here, Figure 1A is a diagram illustrating the overview of the processing according to the processing system 1 according to the embodiment of this disclosure. Specifically, Figure 1A shows an example of the processing in the distribution of video content performed using the processing system 1. According to Figure 1A, the user, as the sender, uses an available sender terminal device to transmit video content featuring the objects of character A and character B to the receiver terminal device of the user, as the receiver. For example, the sender themselves inputs both audio information, which is voice A for character A, and audio information, which is voice B for character B (typically, it is assumed that the sender performs both character A and character B in the video content).

The user, as the recipient, then uses an available recipient terminal device to receive and play video content from the sender terminal device via the server device. However, the recipient may have a need to output only the voice of either character A or character B from the video content being played, depending on their preferences or circumstances, or conversely, to mute the other. In this case, simply changing the volume or audio settings of the playback application would only allow either muting all the audio transmitted from the sender terminal device—that is, both voice A of character A and voice B of character B—or continuously outputting both. However, according to processing system 1, since identification information is pre-assigned to the audio information of character A's voice A and character B's voice B, it is possible to output only the audio desired by the recipient and restrict, i.e., mute, the output of the other. In the example in Figure 1A, only voice A is output, and the output of voice B is restricted, i.e., muted.

This processing system 1 is typically used in video content featuring characters A and B, but it can also be used in other video content such as video conferencing and teleconferencing. In such cases, as described above, it is possible to restrict the output of any of the audio by specifying the character or user identification information of the users participating in the video or teleconferencing.

Thus, in processing system 1, the sender terminal device receives speech information (e.g., audio information for voice A and voice B) associated with each of the multiple objects (e.g., character A and character B) contained in the content (e.g., video content). Meanwhile, the receiver terminal device selects at least one of the multiple objects (e.g., character A). The receiver terminal device then allows the output of speech information (e.g., audio information for voice A) associated with the selected object (e.g., character A), while restricting the output of speech information (e.g., audio information for voice B) associated with other objects (e.g., character B).

In this disclosure, "sender" and "receiver" are merely designations used to distinguish between those who send content and those who receive content. That is, even if someone is described as a sender, they may also be a receiver if they receive content from someone else, and even if someone is described as a receiver, they may also be a sender if they transmit content to someone else. Furthermore, both senders and receivers are not limited to individuals; they may also be organizations such as companies or groups. While this primarily describes cases where the sender generates the content themselves, the sender and the content creator may be different individuals. In this case, even if the content creator only generates the content and does not create the content themselves, if the generated content is transmitted by someone else, they are included as the sender.

Furthermore, in this disclosure, "sender terminal device" and "receiver terminal device" are merely designations used to distinguish between terminal devices that transmit content and terminal devices that receive content. That is, even if a device is described as a sender terminal device, it can become a receiver terminal device if it receives content from another terminal device, and even if a device is described as a receiver terminal device, it can become a sender terminal device if it transmits content to another terminal device.

In this disclosure, "content" means a set of electronic information transmitted and received via a communication network. Examples of such content include video content, music content, game content, publication content, chat content, social networking service (SNS) content, web content, and combinations thereof. Among these, processing system 1 is preferably used for video content that includes at least image information containing multiple character objects as characters, and audio information associated with each character object. In this disclosure, video content includes not only content distributed through video content distribution sites, but also, for example, video conferencing content (including cases where only audio is transmitted and received with the camera function turned off), teleconferencing content, and electronic advertising content such as digital signage. Furthermore, unless otherwise specified, the following explanation will use video content as an example of content; however, content is not necessarily limited to video content.

In this disclosure, "object" means data contained within content or means for manipulating and inputting such data. Examples of such objects include character objects, structural objects, decorative objects, text objects, image objects, GUI objects, and combinations thereof. Among these, the processing system 1 is preferably used for character objects included as characters within video content (for example, character A and character B in Figure 1A). In the following, unless otherwise specified, the example of an object will be a character object; however, objects are not necessarily limited to character objects.

In this disclosure, "processing device" means any of the devices constituting the processing system 1, and may be a server device, a sender terminal device, or a receiver terminal device. Furthermore, the processing device is not limited to any single device, but may be a combination of multiple devices capable of distributing the processing performed by the processing device. "Processing program" and "processing method" mean the program and method executed in the said processing device.

2. Diagram 1B of the configuration of the processing system 1 is a block diagram showing the configuration of the processing system 1 according to one embodiment of the present disclosure. According to Figure 1B, the processing system 1 includes a server device 100 for processing content (for example, video content), a sender terminal device 200-1 for transmitting content, and a receiver terminal device 200-2 for receiving content, and these are connected to each other so as to be able to communicate via a communication network.

In Figure 1B, the sender terminal device 200-1 and the receiver terminal device 200-2 are shown as single devices, but naturally, each may include multiple devices.

Furthermore, although Figure 1B shows a single server device 100, multiple server devices and other devices may be combined to distribute processing and storage. In this case, the server device 100 may include combinations of multiple server devices and other devices.

3. Configuration diagram 2A of the server device 100 is a block diagram showing the configuration of a server device 100 according to one embodiment of the present disclosure. The server device 100 does not need to have all of the components shown in Figure 2A; it is possible to omit some components or add other components. The server device 100 does not need to have all the components shown in Figure 2A in a single enclosure; it is possible to distribute each component and processing of the server device 100 across multiple devices.

According to Figure 2A, the server device 100 includes a processor 111 composed of a CPU, RAM, ROM, and memory 112 including non-volatile memory and an HDD, and a communication interface 113. These components are electrically connected to each other via control lines and data lines.

The processor 111 is composed of a CPU (microcomputer) and functions as a control unit for controlling other connected components based on various programs stored in the memory 112. The processor 111 reads and executes programs for running the application and the OS from the memory 112. Specifically, the processor 111 executes processes such as "receiving speech information from the sender terminal device 200-1 via the communication interface 113, associated with each of the multiple objects included in the content generated in the sender terminal device 200-1," and "transmitting the received content to the receiver terminal device 200-2 via the communication interface 113," based on programs stored in the memory 112. The processor 111 is mainly composed of one or more CPUs, but a GPU, FPGA, etc., may be combined as appropriate.

Memory 112 includes RAM, ROM, non-volatile memory, and HDD, and functions as a storage unit. ROM stores instruction commands for executing the application and OS related to this disclosure as programs. Such programs are loaded and executed by the processor 111. RAM is used to write and read data while the program stored in ROM is being processed by the processor 111. Non-volatile memory is memory where data is written and read as a result of program execution, and data written there is retained even after the program execution has finished. Specifically, memory 112 stores programs for the processor 111 to perform the above-mentioned processing.

The communication interface 113 functions as a communication unit that transmits and receives information with other devices, such as the remotely installed transmitter terminal device 200-1 and receiver terminal device 200-2, via the communication processing circuit and antenna. The communication processing circuit performs processing to transmit and receive programs and various information used in the processing system 1 as processing progresses. The communication processing circuit processes based on a wideband wireless communication method, such as LTE, but can also process based on narrowband wireless communication methods such as wireless LAN (as represented by IEEE 802.11) or Bluetooth®, or contactless wireless communication methods. Furthermore, wired communication can be used in place of, or in addition to, wireless communication.

4. Diagram 2B of the configuration of the terminal device 200 is a block diagram showing the configuration of the terminal device 200 according to one embodiment of the present disclosure. The terminal device 200 does not need to have all of the components shown in Figure 2B; it is possible to omit some components or to add other components. Furthermore, although the terminal device 200 is used as a sender terminal device 200-1 or a receiver terminal device 200-2, the two do not need to have the same configuration; each terminal device may have a different configuration.

According to Figure 2B, the terminal device 200 includes a processor 211 composed of a CPU, RAM, ROM, and memory 212 including non-volatile memory and HDD, a communication interface 213, an input interface 214, and an output interface 215. These components are electrically connected to each other via control lines and data lines.

The processor 211 consists of a CPU (microcomputer) and functions as a control unit for controlling other connected components based on various programs stored in the memory 212. The processor 211 reads and executes programs for running the application and the operating system from the memory 212. The processor 211 is mainly composed of one or more CPUs, but may be combined with a GPU, FPGA, etc., as appropriate.

When the processor 111 functions as the sender terminal device 200-1, it executes the following processes based on a program stored in memory 212: "receiving operation input from the sender via the input interface 214 and starting an application program for generating content," "receiving operation input from the sender via the input interface 214 and selecting one of several objects included in the content based on the inputted identification information," "inputting speech information associated with the selected object via the input interface 214," and "transmitting content including image information and speech information to the server device 100 via the communication interface 213."

Furthermore, when the processor 111 functions as a receiver terminal device 200-2, it executes the following processes based on a program stored in memory 212: "receiving operation input from the receiver via the input interface 214, launching an application program for outputting content, and selecting the desired content"; "receiving content from the sender terminal device 200-1 via the communication interface 213, including speech information input associated with each of the multiple objects included in the content generated at the sender terminal device 200-1"; "outputting the selected content via the output interface 215"; "selecting at least one object from the multiple objects included in the content via the input interface 214"; and "outputting speech information associated with the selected at least one object when outputting speech information included in the content via the output interface 215."

Memory 212 includes RAM, ROM, or non-volatile memory and functions as a storage unit. ROM stores instruction commands for executing the application and OS according to this disclosure as a program. Such programs are loaded and executed by the processor 211. RAM is used to write and read data while the program stored in ROM is being processed by the processor 211. Non-volatile memory is memory in which data is written and read as a result of program execution, and the data written therein is retained even after the program execution has finished. Specifically, memory 212 stores programs for the processor 211 to perform the above-mentioned processing, etc.

The communication interface 213 functions as a communication unit that transmits and receives information between the electrically connected server device 100 and other terminal devices 200 via the communication processing circuit. The communication processing circuit performs processing to transmit and receive programs and various other information used in the processing system 1 as processing progresses. While the communication processing circuit processes based on a broadband wireless communication method such as LTE, it can also process based on narrowband wireless communication methods such as wireless LAN (as represented by IEEE 802.11) or Bluetooth®, or contactless wireless communication methods. Furthermore, wired communication can be used in place of, or in addition to, wireless communication.

The input interface 214 functions as an input unit that accepts operation input from the sender or receiver to the terminal device 200, as well as input of various types of information by the sender or receiver. Examples of the input interface 214 include various hard keys such as keyboards and mice, a touch panel superimposed on the display of a display device and having an input coordinate system corresponding to the display's coordinate system, a microphone for inputting voice information (a type of spoken information), and sensors for sensing the external environment, such as a camera for capturing images. In the case of a touch panel, icons corresponding to the command to be input are displayed on the screen, and the user or business operator makes a selection of each icon by performing operation input via the touch panel. The detection method for operation input via the touch panel can be any method, such as capacitive or resistive. The input interface 214 does not always need to be physically provided on the terminal device 200; it may be connected via a wired or wireless network as needed.

The output interface 215 functions as an output unit for outputting various types of information. An example of the output interface 215 is an interface for connecting to an external device or equipment, such as a display device composed of a liquid crystal panel, organic EL display, or plasma display. However, if the terminal device 200 itself has a display, that display can function as the output interface. Furthermore, if it is connected to a display device via a communication interface 213, the communication interface 213 can also function as the output interface 215.

6. Examples of Content In this embodiment, as described above, content is generated in the sender terminal device 200-1, and the generated content is output to the receiver terminal device 200-2 via the server device 100. Such content includes multiple objects, and speech information is associated with each object. Such content refers to a set of electronic information transmitted and received via a communication network. Examples include video content, music content, game content, publication content, chat content, SNS content, web content, and combinations thereof. Among these, the processing system 1 is preferably used for video content that includes at least image information in which multiple objects, namely character objects, are characters, and audio information associated with each character object. In the following, unless otherwise specified, the explanation will be given using the case where the content is video content as an example, but of course, the content is not limited to video content, and the processing according to this embodiment can be executed similarly even with other types of content.

Figure 3 is a schematic diagram illustrating information transmitted from the sender terminal device 200-1 as transmission information according to one embodiment of this disclosure. Specifically, Figure 3 shows an example of video content that is generated in the sender terminal device 200-1, stored in memory 212, and then transmitted to the server device 100 and stored in the content management table.

According to Figure 3, video content includes image information and audio information, associated with the content ID information of the video content. "Content ID information" is unique to each video content and is used to identify each video content. This content ID information is generated each time new video content is generated at the sender terminal device 200-1, or each time new video content is received at the server device 100.

"Image information" refers to image data that constitutes the video content. This image information may be a still image, a moving image, or a combination thereof. Such image information may be captured from real space by a camera provided as one of the input interfaces 214 in the sender terminal device 200-1, or it may be virtually generated by the processing of the processor 211. The image information includes at least multiple objects that are identifiable from one another, and each object is assigned object ID information. For example, in the example in Figure 3, the image information includes a character object of character A with object ID information "B1" and a character object of character B with object ID information "B2". Examples of such objects include character objects, structural objects, decorative objects, text objects, image objects, GUI objects, and combinations thereof. Among these, the processing system 1 is preferably used for character objects that are included as characters in the video content (for example, character A and character B in Figure 1A). Note that, unless otherwise specified, the following explanation will describe the case of character objects as examples of objects, but of course, objects are not limited to character objects.

"Audio information" is a type of speech information and constitutes audio data that makes up the video content. For example, this audio information is audio data input by a microphone provided as one of the input interfaces 214 in the sender terminal device 200-1, containing the sender's voice, etc. However, the audio information may also be, for example, audio data that reproduces the voice of a character object based on text information input via the input interface 214, text data that transcribes the sender's voice input via the microphone, text data generated based on text information input via the input interface 214, or other data converted from at least one of these. Such audio information is typically stored in association with the object ID information of each object included in the video content. For example, in the example in Figure 3, audio information A is stored in association with "B1," which is the object ID information of character A, audio information B is stored in association with "B2," which is the object ID information of character B, and BGM audio information is stored as audio information not associated with any object ID information.

In other words, as shown in Figure 3, a video content with content ID information "A1" is shown as an example of content. This video content includes image information that is a video consisting of multiple frames from F1 to Fn and having a length from time t0 to time tn. At least one of the frames of this image information includes, as characters, character A with object ID information "B1" and character B with object ID information "B2" as objects. Here, it is assumed that the sender himself is acting out character A and character B respectively using his own sender terminal device 200-1, as illustrated in Figure 1A. Therefore, this video content includes voice information A of character A, which is input started at time t0 and ended at time t2. Also, this video content includes voice information B of character B, which is input started at time t1 and ended at time t4, as a result of receiving operation input from the sender at time t1. Furthermore, the video content includes voice information A of character A, which was inputted at time t3 and ended at time t6, as the sender's input was received at time t3. The video content also includes voice information B of character B, which was inputted at time t4 and ended at time tn, as the sender's input was received at time t5. Additionally, during the period from time t1 to time t6, background music (BGM) audio information is included as audio information not associated with any object. That is, in the example in Figure 3, for example, from time t1 to time t2, from time t3 to time t4, and from time t5 to time t6, voice information A of character A and voice information B of character B are played simultaneously.

Thus, in the case of video content, the content includes image information and audio information, associated with the content ID information. The image information includes each frame (image data) constituting the video, associated with time (e.g., t0 to tn), frame ID information identifying each frame (e.g., F1 to Fn), and object ID information identifying each object contained in at least one of the frames of the image information, synchronized with time (e.g., t0 to tn). The audio information includes each audio data, associated with time (e.g., t0 to tn), and object ID information associated with each audio data (although there may be cases where no associated object ID information exists).

The video content shown in Figure 3 is merely one example of content, as described above. Therefore, even when using video content, it is not necessary to include all of the various types of information exemplified above, and additional information may be included.

Furthermore, as mentioned above, audio information is a type of speech information. Speech information can be associated with the object ID information of an object, and can be any type of information that can reproduce the information entered by the sender. Besides audio information, it can also be text information or image information, for example.

7. Processing Sequence Executed by Processing System 1 Figure 4 is a diagram showing a processing sequence executed by processing system 1 according to one embodiment of the present disclosure. Specifically, Figure 4 is a diagram showing a series of processing sequences from when content is generated in the sender terminal device 200-1 until the generated content is output to the receiver terminal device 200-2 via the server device 100. Processing in each device is executed by a processor processing a program stored in the memory of each device.

(A) Processing related to content generation As shown in Figure 4, first, the processing related to content generation is mainly performed in the sender terminal device 200-1. The processor 211 of the sender terminal device 200-1 receives operation input from the sender via the input interface 214, reads the application program for content generation from the memory 212, and starts the application program (S11). When the application program is started, the processor 211 starts storing the image data, which is frame F1, as image information of the content, as shown in Figure 3. At this time, when the processor 211 receives operation input from the sender via the input interface 214 to request input of audio information associated with an object included in the image information of the content (S12), it outputs an audio input screen via the output interface 215.

Here, Figure 6 shows an example of a screen output in a sender terminal device 200-1 according to one embodiment of the present disclosure. Specifically, Figure 6 shows an example of a voice input screen 10 output when the sender terminal device 200-1 receives an operation input from the sender requesting voice information input in S12 of Figure 4. According to Figure 6, the voice input screen 10 includes the name of the application program, "Application A," along with an image information display area 11 and an object selection area 12. The image information display area 11 outputs image data 13 of frame F1, for example, as currently recorded image information. This image data 13 includes an image 14 of character A and an image 15 of character B as character objects.

Here, character A shown in image 14 is assigned object ID information "B1", and character B shown in image 15 is assigned object ID information "B2". This assignment of object ID information is performed, for example, when video recording is being performed by a camera, by the processor 211 executing object detection processing to detect objects included in each frame, and assigning object ID information to each project at the moment each object is first detected. Furthermore, when, for example, the sender terminal device 200-1 virtually generates image information in a virtual space, the processor 211 assigns object ID information to the objects drawn during that generation. Therefore, in the example of Figure 6, only images 14 of character A and 15 of character B happen to be included, but if images of other characters are newly included, character ID information for those other characters will be generated.

The object selection area 12 contains icons for selecting objects corresponding to objects in the image information that have been assigned object ID information. In the example in Figure 6, the object selection area 12 includes a character A icon 16 corresponding to character A and a character B icon 17 corresponding to character B. When the processor 211 receives an operation input from the sender for any of the icons in the object selection area 12 (for example, either character A icon 16 or character B icon 17) via the input interface 214, it selects the character corresponding to the icon for which the operation input was made. In the example in Figure 6, character A icon 16 is displayed in a way that allows it to be identified from the other icons, indicating that the object ID information of character A (i.e., "B1") has been selected as the object ID information of the character to which the incoming audio information will be associated.

Returning to Figure 4, as shown in Figure 6, when the object ID information of a character for which voice information input is desired is selected, the processor 211 of the sender terminal device 200-1 accepts the voice information input via the input interface 214, associating it with the object ID information (S13). Specifically, the processor 211 stores the voice data spoken by the sender from the microphone, which is one of the input interfaces 214, as voice information in the memory 212, synchronized with each frame of the image information associated with the time (e.g., T0) during which the voice information is being input.

The processor 211 of the sender terminal device 200-1 repeatedly performs the recording of image information, the selection of object ID information to associate with audio information, and the input of audio information as described in S11 to S13, thereby generating content with content ID information "A1," as illustrated in Figure 3. Once content generation is complete, the processor 211 stores the generated content (T11) in memory 212, associating it with the content ID information, and transmits the generated content to the server device 100 via the communication interface 213.

When the processor 111 of the server device 100 receives content from the sender terminal device 200-1, it stores the received content in the content management table (not shown) of memory 112, associating it with the content ID information (S14). Specifically, the processor 111 of the server device 100 stores each piece of information (image information, object ID information, and audio information, etc.) included in the content shown in Figure 3, associating it with the content ID information in the content management table (not shown) of memory 112. With this, the processing related to content generation is completed.

In Figure 4, the processor 211 of the sender terminal device 200-1 transmitted the content to the server device 100 when content generation was completed. However, the content may be divided and transmitted each time a predetermined number of frames or amount of data is generated.

Furthermore, while Figure 4 describes the process assuming simultaneous recording of image information and input of audio information, the processor 211 of the transmitter terminal device 200-1 may first generate the image information and then input the audio information in synchronization with each frame of the image information. For example, in the example shown in Figure 3, audio information A and audio information B are input simultaneously from time t1 to time t2. However, by inputting each audio information in synchronization with each frame after the image information has been generated, the same transmitter can perform both character A and character B.

Furthermore, although not specifically illustrated in Figure 4, as shown in Figure 3, it is also possible to input audio information that is not associated with the object ID information of a specific object (for example, background music audio information).

(B) Processing related to content output Next, as shown in Figure 4, processing related to content output is mainly performed in the receiver terminal device 200-2. This processing involves, for example, if the content is video content, selecting the desired video content in the receiver terminal device 200-2 and playing the video content. The processor 211 of the receiver terminal device 200-2 receives operation input from the receiver via the input interface 214, reads an application program for content output from the memory 212, and starts the application program (S21). Once the application program is started, the processor 211 outputs a content selection screen via the output interface 215, which displays a list of thumbnail images for selecting one or more video content that can be output in the receiver terminal device 200-2. The processor 211 then receives operation input from the receiver via the input interface 214 to select the thumbnail image of the desired content from the list in the content selection screen and selects the content to be output (S22). The processor 211 transmits a content request (T21) to the server device 100 via the communication interface 213, along with content ID information associated with the selected content (for example, content ID information is "A1"), requesting that the content be transmitted.

When the processor 111 of the server device 100 receives a content request from the receiver terminal device 200-2 via the communication interface 113, it refers to the content management table based on the content ID information (e.g., A1) received along with the request and reads the content (e.g., the information illustrated in Figure 3) (S23). The processor 111 then transmits the read content (T22) to the receiver terminal device 200-2 that sent the content request via the communication interface 113.

The processor 211 of the receiver terminal device 200-2 receives content via the communication interface 213 and outputs the received content via the output interface 215 (S24). Here, the processor 211 of the receiver terminal device 200-2 can select the audio information to be output via the receiver's operation input if the received content is associated with an object ID information corresponding to a frame constituting the currently outputting image information, via the input interface 214. That is, when the processor 211 receives operation input from the sender requesting audio information to be input associated with an object (S25), it selects the audio information to be output via the output interface 215 and executes a process to change the output audio information (S26). Details of the series of processes related to S24 to S26 will be described later in Figure 5.

The receiver terminal device 200-2 then repeatedly outputs content, selects objects, and modifies the audio information output according to the selection, as described in S24-S26. When time tn is reached, it terminates the content output. Thus, the process related to content output is completed.

In Figure 4, the processor 211 of the receiver terminal device 200-2 receives the content from the server device 100 as a single data set. However, it may also receive the content in predetermined frame or data chunks and output it sequentially.

8. The processing flow diagram 5 of the receiver terminal device 200-2 is a diagram showing the processing flow executed in the server device 100 according to one embodiment of the present disclosure. Specifically, it is a diagram showing the processing flow related to content output performed by the receiver terminal device 200-2 in steps S24 to S26 of Figure 4. This processing flow is mainly performed by the receiver terminal device 200-2 reading and executing a program stored in the memory 212.

According to Figure 5, the processor 211 receives desired content (for example, content with content ID information A1 as shown in Figure 3) from the server device 100 via the communication interface 213 (S111). Upon receiving the content, the processor 211 outputs it via the output interface 215. Specifically, the processor 211 outputs the image information contained in the received content sequentially from frame F1 via the display, which is one of the output interfaces 215. The processor 211 also outputs the audio information contained in the received content via the speaker, another output interface 215, synchronized with the image information frames. In the example in Figure 3, character A's audio information A is output from time t0, and at time t1, in addition to audio information A, character B's audio information B and BGM audio information are output.

The processor 211, via the input interface 214, can receive operation input from the receiver via the input interface 214 and select the audio information to be output if object ID information is associated with the frame constituting the currently outputting image information in the received content. Therefore, the processor 211 determines whether or not an object has been selected by receiving this operation input (S113).

Here, Figure 7 shows an example of a screen output in a receiver terminal device 200-2 according to one embodiment of the present disclosure. Specifically, Figure 7 shows an example of a content output screen 20 when the receiver terminal device 200-2 receives operation input from a receiver to select the audio information to be output in steps S112 to S113 of Figure 5. According to Figure 7, the content output screen 20 includes the name of the application program, "Application B," along with an image information display area 21 and an object selection area 22. The image information display area 21 displays image data 23 of frame F3, for example, as the currently output image information. This image data 23 includes an image 24 of character A and an image 25 of character B as character objects. Character A, shown in image 24, is assigned the object ID information "B1," and character B, shown in image 25, is assigned the object ID information "B2."

The object selection area 22 contains icons for selecting each object, i.e., character, corresponding to the object ID information associated with the audio information synchronized with the currently output image frame. For example, in the content output screen 20 at either time t1 or t2 in Figure 3, both audio information A for character A and audio information B for character B are output at that time. Therefore, the object selection area 22 includes a character A icon 26 corresponding to character A and a character B icon 27 corresponding to character B. When the processor 211 receives a user input via the input interface 214 for any of the icons in the object selection area 22 (for example, either character A icon 26 or character B icon 27), it selects the character corresponding to the icon for which the user input was made. In the example in Figure 7, character A icon 26 is displayed identifiable from the other icons, indicating that the object ID information of character A (i.e., "B1") has been selected as the object ID information of the character to which the incoming audio information will be associated.

In the example shown in Figure 7, the object selection area 22 includes an icon for selecting the audio information to be output, corresponding to the character included in the currently outputting frame. However, this is not limited to this example. For characters appearing in at least one frame throughout the entire content, an icon for selecting audio information may always be included, allowing for audio information selection even when no audio information is being output.

Furthermore, Figures 5 (S113) and 7 illustrate the case where an object is selected by receiving operation input from the receiver via the input interface 214. However, it is also possible to select objects using sensors such as microphones or cameras as the input interface 214, either as an alternative or in addition to this. For example, the processor 211 uses the camera to recognize the attributes of the receiver using the receiver terminal device 200-2 (e.g., age, gender, etc.). The processor 211 then selects objects for audio output based on this recognition result. For example, a digital signage terminal device could be provided as the receiver terminal device 200-2, and the camera mounted on the terminal device could recognize the attributes of the user (receiver) viewing the terminal device's display. If the user (receiver) is recognized as a "child," audio from objects other than child-oriented objects (e.g., animal characters) would be muted; if the user is recognized as an "adult," audio from objects other than adult-oriented objects (e.g., human characters) would be muted. In this way, using sensors such as cameras as the input interface 214 makes it possible to realize a wider variety of selection methods.

Returning to Figure 5, as shown in Figure 7, when the object ID information of a character whose audio information output is desired is selected, the processor 211 outputs only the audio information of the selected character and restricts (e.g., mutes) the output of audio information of other characters (S114). That is, between time t1 and time t2 in Figure 3, if the object ID information of character A is selected, the processor 211 restricts the output from the output interface 215 (e.g., speaker) of character B's audio information B, so that only character A's audio information A is output. On the other hand, if no object is selected in S113, the process related to S114 is skipped.

Processor 211 continuously repeats the processes described in S112 to S114 until it has finished outputting the series of content from time t0 to tn. This completes the processing flow.

In this explanation, we are describing a scenario where only the voice information A for character A and the voice information B for character B are included in the content. Therefore, when character A is selected, the output of voice information B is restricted, and only voice information A is output. However, in Figure 7, the output of the selected character's voice information may be restricted, while the voice information of the unselected character is output without restriction.

Furthermore, if the content contains three or more audio pieces,
Audio information can be output in various combinations, such as (1) outputting only the audio information of one selected character and restricting the output of audio information for all remaining characters, (2) restricting the output of audio information for one selected character and outputting audio information for all remaining characters, (3) outputting audio information for multiple selected characters and restricting the output of audio information for all remaining characters, or (4) restricting the output of audio information for multiple selected characters and outputting audio information for all remaining characters.

Furthermore, while the above example used "muting" as a method for restricting audio output, various other restriction methods can be employed. These include changing the output volume (for example, lowering it), simultaneously outputting subtitle text information with normally outputted audio, but not outputting subtitles for restricted audio.

In this embodiment, it is possible to provide a processing device, processing program, and processing method that are more user-friendly for users such as receivers. In particular, when the output content includes multiple speech information (e.g., voice information), it is possible for the receiver to select which speech information (e.g., voice information) to output. For example, conventionally, if the receiver did not want to output voice information associated with certain objects, the output was limited by controlling the volume using a volume button on the receiver terminal device 200-2, etc. Therefore, the output of all voice information was limited. However, in this embodiment, it is possible to selectively output only the voice information associated with the object desired by the receiver at the timing desired by the receiver, or to selectively limit the output.

9. Modifications Below are modifications of the above embodiment shown in Figures 1 to 7. Note that the following modifications and the embodiment shown in Figures 1 to 7 can be combined and implemented. Furthermore, except for points specifically mentioned below, the process can be carried out in the same manner as described in the embodiment shown in Figures 1 to 7.

(A) Variation 1 of the selection of audio information
In the above, as shown in Figure 4, etc., the case described is one in which the audio information associated with the character selected in the receiver terminal device 200-2 is selected by the processor 211 of the receiver terminal device 200-2, and the output of audio information for other characters is restricted. However, instead, the audio information associated with the character selected in the receiver terminal device 200-2 may be reorganized by the processor 111 of the server device 100, and the output of audio information for other characters may be restricted.

Figure 8A is a diagram showing a processing sequence executed in a processing system 1 according to one embodiment of this disclosure. Specifically, Figure 8A shows a processing sequence when the processing related to the selection of speech information, which is one of the speech information, is performed by the processor 111 of the server device 100. Processing in each device is executed by the processor processing a program stored in the memory of each device.

Note that the processes related to content generation in S31 to S34 are the same as the processes related to content generation in S11 to S14 shown in Figure 4, therefore their explanation is omitted.

Furthermore, regarding the content output process, the processes S41 to S45, which involve receiving the content output, launching the application program, outputting the desired content via the output interface, and selecting the audio information associated with the desired object, are the same as the processes S21 to S25 shown in Figure 4 regarding the content output process; therefore, their explanation is omitted.

When the object ID information of a character from which audio information output is desired is selected using the method shown in Figure 7, the processor 211 of the receiver terminal device 200-2 transmits object selection information (T43), which includes the content ID information of the content to be output and the selected object ID information, to the server device 100 via the communication interface 213.

When the processor 111 of the server device 100 receives object selection information, it reads the content associated with the content ID information from the memory 112 and performs a process to reorganize the content (S46). Specifically, the processor 111 keeps the audio information associated with the selected object ID information (audio information A in the example of Figure 7) as is, and deletes the other audio information that was not selected (audio information B in the example of Figure 7) from the content. After reorganizing the content through the above process, the processor 111 stores the reorganized content in the memory 112 and transmits the content (T44) to the receiver terminal device 200-2 that sent the object selection information via the communication interface 113.

When the receiver terminal device 200-2 receives the reorganized content via the communication interface 213, it outputs the received content via the output interface 215, similar to S44. At this time, the audio information of the content does not include the audio information B of character B. Therefore, the processor 111 of the receiver terminal device 200-2 outputs only the audio information A of character A via the output interface 215, without outputting the audio information B of character B.

Furthermore, for example, in the content output screen 20 output in Figure 7, the object selection area 22 will always display icons corresponding to objects (characters) included in at least some of the frames of the image information. This ensures that even if the output of audio information B is restricted, the recipient can select audio information B again if they wish to receive it.

As shown in Figure 8A, selective output of audio information is possible, similar to the embodiments in Figures 1 to 7.

(B) Modification 2 of the selection of audio information
In the above, as shown in Figure 4, etc., the case described is one in which the audio information associated with the character selected in the receiver terminal device 200-2 is selected by the processor 211 of the receiver terminal device 200-2, and the output of audio information for other characters is restricted. However, instead, the audio information associated with the character selected in the receiver terminal device 200-2 may be selected by the processor 211 of the sender terminal device 200-1, and the output of audio information for other characters may be restricted.

Figure 8B is a diagram showing a processing sequence executed in a processing system 1 according to one embodiment of this disclosure. Specifically, Figure 8B shows a processing sequence when the processing related to the selection of voice information, which is one of the speech information, is performed by the processor 211 of the sender terminal device 200-1. Processing in each device is executed by the processor processing a program stored in the memory of each device.

Except for the fact that the content is streamed in fixed data chunks, the processes related to content generation S61-S64 are the same as the processes related to content generation S11-S14 shown in Figure 4, so their explanation will be omitted.

Furthermore, except for the fact that the content is streamed in fixed data chunks, the processing related to content output, specifically steps S71-S75, which involve receiving the content output, launching the application program, outputting the desired content via the output interface, and selecting the audio information associated with the desired object, is the same as the processing related to content output shown in Figure 4, specifically steps S21-S25. Therefore, a detailed explanation of these steps is omitted.

When the object ID information of a character from which audio information output is desired is selected using the method shown in Figure 7, the processor 211 of the receiver terminal device 200-2 transmits object selection information (T43), which includes the content ID information of the content to be output and the selected object ID information, to the server device 100 via the communication interface 213.

When the processor 111 of the server device 100 receives object selection information (T73), it identifies the sender terminal device 200-1, which is the sender of the content, based on the content ID information (S76). Then, the processor 111 transmits the object selection information (T74) to the identified sender terminal device 200-1 via the communication interface 113.

The processor 211 of the sender terminal device 200-1, upon receiving object selection information via the communication interface 213, selectively performs input of audio information (S77). Specifically, while the sender terminal device 200-1 inputs and distributes image and audio information in real time, the processor 211 accepts input of audio information associated with object ID information through the processes shown in S62 and S63 (i.e., the processes shown in S22 and S23 of Figure 4). The processor 211 then refers to the object ID information received via the object selection information, and if audio information associated with the same object ID information is input, it stores that audio information synchronized with the image information. On the other hand, the processor 211 accepts input of audio information associated with object ID information different from the received object ID information, but does not include it in the transmitted content. That is, the processor 211 generates content that includes only audio information associated with the object ID information of the character selected by the receiver, and does not include audio information associated with the object ID information of other characters.

The processor 211 of the sender terminal device 200-1 transmits the content (T75) generated as described above, along with the content ID information, to the server device 100 via the communication interface 213. Upon receiving the content via the communication interface 113, the processor 111 of the server device 100 stores it in the content management table, associating it with the content ID information, and also transmits the received content (T76) to the receiver terminal device 200-2, which sent the object selection information, via the communication interface 113.

The processor 211 of the receiver terminal device 200-2, upon receiving content via the communication interface 213, outputs the received content via the output interface 215. At this time, the received content includes only the audio information associated with the object ID information of the selected character, as described above, and does not include audio information associated with the object ID information of other characters. In other words, audio information associated with object ID information of characters other than the one selected by the receiver is restricted from transmission, thereby limiting its output in the receiver terminal device 200-2.

As described above, the example shown in Figure 8B also enables selective output of audio information, similar to the embodiments shown in Figures 1 to 7.

(C) Modifications relating to restricted audio information Figures 1 to 8B illustrate the case where only audio information A for character A and audio information B for character B are included in the content. Therefore, when character A is selected, the output of audio information B is restricted, and only audio information A is output. However, it is also possible to restrict the output of the audio information of the selected character and output the audio information of the other character without restriction.

Furthermore, if the content contains three or more audio pieces,
Audio information can be output in various combinations, such as (1) outputting only the audio information of one selected character and restricting the output of audio information for all remaining characters, (2) restricting the output of audio information for one selected character and outputting audio information for all remaining characters, (3) outputting audio information for multiple selected characters and restricting the output of audio information for all remaining characters, or (4) restricting the output of audio information for multiple selected characters and outputting audio information for all remaining characters.

(C) Modification with Multiple Transmitters In the examples in Figures 1 to 8B, the case in which one sender plays multiple characters was explained by inputting the voice information of multiple characters in association with object ID information in one sender terminal device 200-1. However, instead of this, or in addition to this, it is also possible for multiple senders to play the same character, or for multiple senders to play multiple characters, by inputting the voice information of multiple characters in association with object ID information in multiple sender terminal devices 200-1.

Figure 9 is a diagram illustrating an overview of the processing according to the processing system 1 according to the embodiment of this disclosure. Specifically, Figure 9 shows an example of processing in the distribution of video content performed using the processing system 1. According to Figure 9, for the same video content, the sender terminal device of sender A receives voice information A for character A and voice information B for character B, and transmits them to the receiver terminal device of the recipient via the server device. In addition, the sender terminal device of sender B receives voice information C for character C and voice information D for character D, and transmits them to the receiver terminal device of the recipient via the server device. At this time, sender ID information for identifying sender A or sender terminal device of sender A is associated with voice information A and voice information B. In addition, sender ID information for identifying sender B or sender terminal device of sender B is associated with voice information C and voice information D. Therefore, when selecting the voice information to be output in the receiver terminal device, it is possible to select sender ID information instead of object ID information. For example, if the receiver terminal device selects the sender ID information of sender A, only voice information A and voice information B will be output, and the output of voice information C and voice information D will be restricted. Similarly, if the receiver terminal device selects the sender ID information of sender B, only voice information C and voice information D will be output, and the output of voice information A and voice information B will be restricted.

As shown in Figure 9, selective output of audio information is possible, similar to the embodiments in Figures 1 to 8B.

(D) Modifications relating to content, objects, and speech information In the examples in Figures 1 to 8B, video content was used as an example of content, and the explanation was given using the case where the object is a character object and the speech information is audio information. However, regardless of whether the content is video content or other content, similar processing is possible for other objects and other speech information. For example, in addition to video content, content can include music content, game content, publication content, chat content, SNS content, web content, and combinations thereof. Similarly, in addition to character objects, objects can include structure objects, decorative objects, text objects, image objects, GUI objects, and combinations thereof. Furthermore, in addition to audio information, speech information can include text information, image information, and combinations thereof.

For example, when applying chat content as content to the embodiments described herein, the objects include GUI objects in the shape of speech bubbles associated with each sender, and the utterance information includes text information entered by each user as chat. Even in such a case, the recipient can restrict the output (display) of chat (text information) associated with the GUI objects of other senders by selecting the GUI object of the desired sender. This makes it possible to selectively output only specific senders.

The processes and procedures described herein can be implemented not only by those explicitly described in the embodiments, but also by software, hardware, or a combination thereof. Specifically, the processes and procedures described herein are implemented by implementing the logic corresponding to the process on a medium such as an integrated circuit, volatile memory, non-volatile memory, magnetic disk, or optical storage. Furthermore, the processes and procedures described herein can be implemented as computer programs and executed by various computers, including processing units and server devices.

Even if the processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software programs, multiple components, and/or multiple modules. Similarly, even if the various types of information described herein are described as being stored in a single memory or storage unit, such information may be distributed and stored in multiple memories within a single device or in multiple memories distributed across multiple devices. Furthermore, the software and hardware elements described herein may be implemented by integrating them into fewer components or by decomposing them into more components.

1 Processing System 100 Server Device 200 Terminal Device 200-1 Sender Terminal Device 200-2 Receiver Terminal Device

Claims (10)

A processing unit comprising at least one processor,
The at least one processor is
The system receives speech information from the transmitting terminal device via a communication interface, which is input in association with each of the multiple objects included in the content generated in the transmitting terminal device.
Select at least one of the multiple objects via the input interface.
When outputting the utterance information via the output interface, the utterance information associated with the selected at least one object is output.
A processing unit configured to perform a process for that purpose. The aforementioned content is video content,
The aforementioned object is a character object included in the video content.
The apparatus according to claim 1. The processing apparatus according to claim 2, wherein the utterance information is voice information input in association with the character object. The at least one processor is
The system receives each utterance information input associated with each of the aforementioned multiple objects,
When outputting the speech information via the output interface, the output of speech information associated with objects other than the selected at least one object is restricted, thereby limiting the output of speech information associated with the selected at least one object.
The apparatus according to claim 1, configured to perform a process for that purpose. The at least one processor is
Of the speech information input in association with the aforementioned multiple objects, only the speech information of the selected at least one object is received.
When outputting the speech information via the output interface, only the speech information associated with the at least one received object is output.
The apparatus according to claim 1, configured to perform a process for that purpose. The processing apparatus according to claim 5, wherein speech information associated with objects other than the selected at least one object is restricted from being transmitted from the sender terminal device to the processing apparatus. The aforementioned speech information is received from the transmitting terminal device via a server device installed remotely.
The processing apparatus according to claim 5, wherein speech information associated with objects other than the selected at least one object is restricted from being transmitted from the server device to the processing apparatus. The processing device according to claim 1, wherein the speech information is input in association with each of the multiple objects by the sender selecting one of the multiple objects in advance at the sender terminal device. In a computer comprising at least one processor, the at least one processor is:
The system receives speech information from the transmitting terminal device via a communication interface, which is input in association with each of the multiple objects included in the content generated in the transmitting terminal device.
Select at least one of the multiple objects via the input interface.
When outputting the utterance information via the output interface, the utterance information associated with the selected at least one object is output.
A processing program that is designed to perform a specific action. A computer comprising at least one processor, wherein a processing method is performed by the at least one processor,
The process involves receiving speech information from the sender terminal device via a communication interface, which is input in association with each of the multiple objects included in the content generated in the sender terminal device,
A step of selecting at least one object from the plurality of objects via an input interface,
When outputting the speech information via the output interface, the steps include outputting the speech information associated with the selected at least one object,
A processing method that includes this.