JP7465186B2 - Live Streaming System - Google Patents

Description

The present invention relates to a live distribution system.

Recently, due to the effects of infectious diseases and other factors, it has become difficult to hold live performances with audiences, so professional artists are increasingly streaming live performances without an audience.

For such live streaming without an audience, a service is being offered that combines video of the artist with video of the audience watching online (see non-patent document 1).

On the other hand, a service has become popular in which a singer is filmed with a camera installed in the karaoke machine, and the resulting video data is edited together with the singer's voice data while singing, creating a video that can then be uploaded to a server, making the video available to third parties.

For example, Patent Document 1 discloses a technology that broadcasts the entirety of a karaoke live performance in real time.

"Tomorrowland", [online], [searched on September 24, 2020], Internet <URL: https://natalie.mu/music/column/388771/page/2>

JP 2004-029459 A

The object of the present invention is to provide a new live distribution system that can combine and distribute the distributor's voice and video with the viewer's voice and video when live distribution is performed using a karaoke device.

One invention for achieving the above object is a live distribution system in which a plurality of karaoke machines and a server device are communicatively connected, wherein one karaoke machine has a distribution data processing unit that generates distribution data based on the voice of a distributor wishing to live distribute karaoke singing, which is collected by a sound collection means of the one karaoke machine, and a video of the distributor, which is photographed by a filming means of the one karaoke machine, and transmits the generated distribution data to the server device, and a distributor side setting unit that sets the distributor so that the voice and video can be viewed based on the live data distributed from the server device, and another karaoke machine different from the one karaoke machine has a viewer side setting unit that sets the viewer so that the voice and video can be viewed based on the live data distributed from the server device, and a viewer side setting unit that sets the viewer's voice collected by the sound collection means of the other karaoke machine and/or the other karaoke machine so that the viewer side setting unit sets the viewer side setting unit ... The live distribution system includes a determination unit that determines whether the viewer's excitement satisfies a predetermined condition based on an image of the viewer captured by a filming means of the karaoke device, and a viewer data processing unit that generates viewer data based on the viewer's voice and the image of the viewer and transmits the generated viewer data to the server device, and if the excitement does not satisfy the predetermined condition, the viewer data processing unit stops transmitting the viewer data to the server device. The server device includes a live data generation unit that generates live data based on distribution data received from the one karaoke device, and when viewer data is received from the other karaoke device, generates live data by combining the viewer data with the distribution data, and a distribution processing unit that distributes the generated live data to the one karaoke device and the other karaoke devices.
Other features of the present invention will become apparent from the following specification and drawings.

According to the present invention, when live streaming is performed using a karaoke device, the broadcaster's audio and video can be combined with the viewer's audio and video for streaming.

FIG. 1 is a diagram illustrating a live distribution system according to an embodiment. FIG. 1 is a diagram showing a karaoke device K1 according to an embodiment. FIG. 2 is a diagram showing a karaoke main body of the karaoke device K1 according to the embodiment. FIG. 2 is a diagram showing a karaoke main body of a karaoke device K2 according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a server device according to an embodiment. FIG. 4 is a diagram showing a table stored in a storage unit of the server device according to the embodiment. FIG. 2 is a diagram showing the layout of videos of a distributor and viewers according to an embodiment. FIG. 2 is a diagram showing the layout of videos of a distributor and viewers according to an embodiment. 1 is a flowchart showing a process of a live distribution system according to an embodiment. 1 is a flowchart showing a process of a live distribution system according to an embodiment. FIG. 13 is a diagram illustrating a server device according to a modified example. FIG. 11 is a diagram showing a table stored in a storage unit of a server device according to a modified example.

==Live Streaming System==
A live distribution system according to an embodiment and a modification will be described with reference to Fig. 1 to Fig. 12. The live distribution system is a system for distributing live data.

Live data is data for playing audio and video of a distributor and a viewer on a karaoke device. A distributor is someone who wishes to broadcast karaoke singing live. A viewer is someone who listens to audio and video based on the broadcasted live data. Live data is generated on the server device S based on data for broadcast, or data for broadcast and viewer data (details will be described later).

The live distribution system includes multiple karaoke machines and a server device. As shown in FIG. 1, the live distribution system 1 according to this embodiment includes multiple karaoke machines K1 to Kn, and a server device S. Each karaoke machine and the server device S are connected so that they can communicate with each other. Each karaoke machine has a function for performing karaoke and singing karaoke songs.

Here, karaoke device K1 is used by distributor U1 to perform live distribution of karaoke singing. Meanwhile, karaoke devices K2 to Kn are used by viewers U2 to Un to listen to audio and video, respectively. That is, in this embodiment, karaoke devices K2 to Kn are not used for karaoke performance and singing. Karaoke device K1 corresponds to "one karaoke device," and karaoke devices K2 to Kn correspond to "other karaoke devices." Each karaoke device may be installed in a room of one karaoke store, or may be installed in a different karaoke store.

The server device S is a computer that manages various information and information sent from each karaoke device, and distributes live data to each karaoke device.

Karaoke device K1
As shown in FIG. 2, the karaoke device K1 includes a karaoke main unit 10, a speaker 20, a display device 30, a microphone 40, a remote control device 50, and an image capturing means 60.

The karaoke main unit 10 performs various controls related to karaoke performance and karaoke singing, such as karaoke performance control of the selected song, display control of lyrics and background images, and processing of audio signals input through the microphone 40. The speaker 20 is configured to emit sound based on the sound emission signal from the karaoke main unit 10. The display device 30 is configured to display videos and images on the display screen based on the signal from the karaoke main unit 10. The microphone 40 is configured to convert the distributor's conversation voice and the singing voice of the karaoke singer into analog audio signals and input them to the karaoke main unit 10. The microphone 40 is an example of a "sound collection means (of one karaoke device)". The remote control device 50 is a device for performing various operations on the karaoke main unit 10. The shooting means 60 is a camera that can shoot the distributor.

As shown in FIG. 3, the karaoke machine 10 according to this embodiment includes a storage unit 10a, a communication unit 10b, an input unit 10c, a performance unit 10d, and a control unit 10e. Each component is connected to a bus B via an interface (not shown).

[Storage Means]
The storage means 10a is a large-capacity storage device for storing various data, such as a hard disk drive, etc. The storage means 10a stores music data.

The song data is provided with song identification information for identifying each song. The song identification information is information unique to each song, such as a song ID for identifying the song. The song data includes accompaniment data, reference data, etc. The accompaniment data is data that is the source of the karaoke performance sound. The reference data is data that indicates the main melody of the song that is performed karaoke. The storage means 10a also stores background video data such as background video displayed on the display device 30 during karaoke performance, and song attribute information (singer name, genre, etc.).

[Communication means/input means]
The communication means 10b provides an interface for communicating with the server device S and the remote control device 50. The input means 10c is configured to allow the distributor to input various instructions. The input means 10c is a button or the like provided on the karaoke main unit 10. Alternatively, the remote control device 50 may function as the input means 10c.

[Playing means]
The performance means 10d, under the control of the control means 10e, performs karaoke performance of music pieces and processes audio signals input through the microphone 40. The performance means 10d includes a sound source, a mixer, an amplifier, etc. (none of which are shown).

[Control Means]
The control means 10e performs various controls in the karaoke device K1. The control means 10e includes a CPU and a memory (neither of which are shown in the figures). The CPU executes programs stored in the memory to realize various functions.

In this embodiment, the CPU of the karaoke device K1 executes a program stored in the memory, and the control means 10e functions as a live distribution reservation unit 100, a distribution data processing unit 200, and a distributor side setting unit 300.

(Live Streaming Reservation Department)
The live distribution reservation unit 100 performs reservation processing for live distribution.

The distributor inputs distribution information related to the live distribution via the remote control device 50. The distribution information includes, for example, an introductory text about the content of the distribution, the start and end times of the distribution, and the songs to be sung by karaoke.

The live streaming reservation unit 100 assigns a dedicated identifier (hereinafter, "streaming identifier") to the streaming information entered by the streamer, and stores it in a streaming reservation table in the storage unit 10a. The live streaming reservation unit 100 also links the device identifier of the karaoke device, the streaming identifier, and the streaming information corresponding to the streaming identifier, and transmits the linked information to the server device S via the communication unit 10b.

Hereinafter, it is assumed that the live streaming reservation unit 100 assigns a streaming identifier ID_D1 to the streaming information D1 of the live performance L1 input by the streamer U1 and stores it in the streaming reservation table in the storage means 10a. It is also assumed that the live streaming reservation unit 100 links the device identifier ID_K1 of the karaoke device K1, the streaming identifier ID_D1, and the streaming information D1, and transmits them to the server device S via the communication means 10b. It is also assumed that the streaming information D1 includes an introduction to the streaming content, the start and end times of the streaming (19:00-20:00), and the songs to be sung at karaoke (songs X1 to X3).

(Distribution data processing section)
The distribution data processing unit 200 generates distribution data based on the voice of a distributor wishing to live-distribute karaoke singing, collected by the sound collection means of one of the karaoke devices, and video of the distributor, captured by the filming means of one of the karaoke devices, and transmits the generated distribution data to a server device.

The data for distribution includes an audio signal corresponding to the distributor's voice and a video signal corresponding to the distributor's video.

When live streaming, it is common for the streamer to start with a greeting (such as a self-introduction) and then begin singing karaoke. That is, the data for streaming may include an audio signal corresponding to the voice of the greeting, or an audio signal corresponding to the singing voice of the karaoke performance. Also, when singing karaoke, the audio signal may also include the sound of the karaoke performance.

Specifically, suppose that broadcaster U1 has booked the broadcast of live broadcast L1. When the start time of broadcast of live broadcast L1 approaches, the broadcast data processing unit 200 notifies broadcaster U1 of this. The notification can be made in various ways. For example, the broadcast data processing unit 200 can display a message such as "The time for the booked live broadcast is approaching. Please get ready" on the display screen of the display device 30 or remote control device 50.

When the start time of distribution arrives, the distribution data processing unit 200 starts collecting sound with the microphone 40 of the karaoke device K1 and filming the distributor with the imaging means 60. The distributor U1 first gives a greeting. In this case, the distribution data processing unit 200 synthesizes an audio signal corresponding to the voice of the distributor U1's greeting collected by the microphone 40 and a video signal corresponding to the video of the distributor U1 filmed by the imaging means 60, to generate distribution data. The distribution data processing unit 200 transmits the generated distribution data to the server device S.

After the greeting is finished, distributor U1 issues an instruction to start the karaoke performance of song X1 via remote control device 50. Karaoke device K1 reads the accompaniment data of song X1 from storage means 10a, and starts the karaoke performance of song X1 by controlling performance means 10d. In addition, when an instruction to start the karaoke performance of the song is issued, distribution data processing unit 200 transmits a performance start signal to server device S.

The distributor U1 sings karaoke along with the karaoke performance. In this case, the distribution data processing unit 200 synthesizes an audio signal corresponding to the singing voice of the distributor U1 and the karaoke performance sound of the song X1 collected by the microphone 40 with a video signal corresponding to the video of the distributor U1 captured by the imaging means 60, to generate new distribution data. The distribution data processing unit 200 transmits the generated new distribution data to the server device S.

In this way, the distribution data processing unit 200 continuously generates and transmits distribution data from the start time included in the distribution information D1 until the end time arrives (i.e., until the live L1 ends).

(Distributor side setting unit)
The distributor setting section 300 performs setting based on the live data distributed from the server device S so that the distributor can view audio and video.

The distributor setting unit 300, for example, displays video based on the live data of the live show L1 on the display screen of the display device 30, and emits audio based on the live data from the speaker 20. In this case, the distributor U1 can view the video and audio of the live show L1 that he or she is performing. Details of the live data distributed from the server device S will be described later.

==Karaoke machine K2 to karaoke machine Kn==
As described above, the karaoke devices K2 to Kn are used by the viewers U2 to Un to listen to the audio and video of the live broadcast, respectively. Since the karaoke devices K2 to Kn have the same configuration, the following description will be given using the karaoke device K2 as an example. Detailed description of the configuration that is the same as the karaoke device K1 will be omitted.

Like the karaoke device K1, the karaoke device K2 includes a karaoke main unit 10, a speaker 20, a display device 30, a microphone 40, a remote control device 50, and a photographing means 60.

The microphone 40 is configured to convert the audience's voices (e.g., cheers, shouts, applause) into analog audio signals and input them to the karaoke main unit 10. The microphone 40 is an example of a "sound collection means (of another karaoke device)." The imaging means 60 is a camera capable of imaging the audience.

As shown in FIG. 4, the karaoke machine 10 according to this embodiment includes a storage unit 10a, a communication unit 10b, an input unit 10c, a performance unit 10d, and a control unit 10e. Each component is connected to a bus B via an interface (not shown).

[Control Means]
The control means 10e performs various controls in the karaoke device K2. The control means 10e includes a CPU and a memory (neither of which are shown in the figures). The CPU executes programs stored in the memory to realize various functions.

In this embodiment, the CPU of the karaoke device K2 executes a program stored in the memory, and the control means 10e functions as a live viewing reservation unit 400, a viewer side setting unit 500, a determination unit 600, and a viewer data processing unit 700.

(Live Viewing Reservation Department)
The live viewing reservation unit 400 performs reservation processing for viewing a live broadcast.

Suppose a viewer issues an instruction to view a live show being distributed in the live distribution system 1 via the remote control device 50. Based on the instruction, the live viewing reservation unit 400 acquires the distribution identifier and distribution information of the live show scheduled to be distributed from the server device S. Based on the acquired distribution identifier and distribution information, the live viewing reservation unit 400 displays information about the live show being distributed (e.g., the distribution identifier, an introduction to the content of the distribution, and the start time) on the display screen of the display device 30. The viewer selects the live show they wish to view via the remote control device 50.

The live viewing reservation unit 400 links the distribution identifier of the live performance selected by the viewer with the device identifier of the karaoke device used by the viewer, and transmits the link to the server device S via the communication means 10b.

Hereinafter, it is assumed that the live viewing reservation unit 400 of the karaoke device K2 links the device identifier ID_K2 of the karaoke device K2 to the distribution identifier ID_D1 of the live L1 selected by the viewer U2, and transmits the link to the server device S via the communication means 10b. Similarly, it is assumed that the live viewing reservation units 400 of the karaoke devices K3 to K5 link the device identifiers ID_K3 to ID_K5 of the karaoke devices K3 to K5 to the distribution identifier ID_D1 of the live L1 selected by the viewers U3 to U5, respectively, and transmit the link to the server device S via the communication means 10b.

(Viewer side setting unit)
The viewer side setting section 500 performs setting based on the live data distributed from the server device S so that the viewer can view the audio and video.

The viewer setting unit 500, for example, causes the display screen of the display device 30 to display video based on the live data of the live event L1, and causes the speaker 20 to emit audio based on the live data. In this case, the viewer U2 can view the video and audio of the live event L1 being distributed. Details of the live data distributed from the server device S will be described later.

(Determination unit)
The judgment unit 600 judges whether the excitement of the viewer satisfies a predetermined condition based on the viewer's voice collected by the sound collection means of the other karaoke device and/or the image of the viewer captured by the image capture means of the other karaoke device.

The excitement level can be determined at various times. For example, the determination unit 600 can determine the excitement level from the timing when the first karaoke performance starts after the start of the live performance. The determination unit 600 can also determine the excitement level of the audience at predetermined intervals. The predetermined interval is set in advance, such as every 10 seconds after the start of the live performance or every 15 seconds after the karaoke performance of a song starts.

For example, in the above example, after the start of the live performance L1, the distributor U1 instructs the karaoke performance of the song X1 via the remote control device 50 of the karaoke device K1. In this case, the control means 10e of the karaoke device K1 controls the performance means 10d to start the karaoke performance of the song X1. At this time, the karaoke device K1 notifies the server device S of the start of the karaoke performance of the song X1. The server device S transfers the notification to each karaoke device that is distributing the live performance L1.

After receiving the notification, the determination unit 600 can perform a determination of the excitement at predetermined intervals.

The excitement level of a viewer can be determined by various methods. For example, the determination unit 600 can determine the excitement level of a viewer based on the sound pressure level of a specific frequency band detected from the viewer's voice and/or the movement of the viewer detected from the video of the viewer.

Specifically, the determination unit 600 extracts a frequency band corresponding to a specific frequency band (e.g., applause or cheering) from the voice of the viewer U2 collected by the microphone 40, and detects the average sound pressure level over a predetermined period of time. The determination unit 600 determines whether the detected average sound pressure level is equal to or greater than a predetermined sound pressure level. If the detected average sound pressure level is equal to or greater than the predetermined sound pressure level, the determination unit 600 determines that the excitement satisfies a predetermined condition. "(The detected average sound pressure level is) equal to or greater than a predetermined sound pressure level" is an example of a predetermined condition.

Alternatively, the determination unit 600 detects the tempo of the movements of the viewer U2 over a predetermined period of time from the video of the viewer U2 captured by the imaging means 60. The determination unit 600 determines whether the difference between the detected tempo and the tempo of the music output from the speaker 20 is equal to or less than a predetermined tempo. If the difference between the detected tempo and the tempo of the music output from the speaker 20 is equal to or less than the predetermined tempo, the determination unit 600 determines that the excitement satisfies a predetermined condition. "(The difference between the detected tempo and the tempo of the music output from the speaker 20) is equal to or less than a predetermined tempo" is an example of a predetermined condition.

The determination unit 600 may determine that the excitement satisfies a predetermined condition when the detected average sound pressure level is equal to or higher than a predetermined sound pressure level and when the difference between the detected tempo and the tempo of the music output from the speaker 20 is equal to or lower than the predetermined tempo. In this case, the predetermined condition corresponds to "(the detected average sound pressure level is equal to or higher than a predetermined sound pressure level)" and "(the difference between the detected tempo and the tempo of the music output from the speaker 20 is equal to or lower than the predetermined tempo)."

(Viewer data processing unit)
The viewer data processing unit 700 generates viewer data based on the viewer's voice and the viewer's video, and transmits the generated viewer data to the server device.

The viewer data processing unit 700 generates viewer data based on the viewer's voice collected by the microphone 40 of the other karaoke device and the image of the viewer captured by the imaging means 60 of the other karaoke device, and transmits the generated viewer data to the server device.

The viewer data includes an audio signal corresponding to the viewer's voice and a video signal corresponding to the viewer's video.

Specifically, assume that viewer U2 is viewing a live performance L1 distributed by karaoke device K2. When the start time of the distribution arrives, viewer data processing unit 700 starts collecting sound with microphone 40 of karaoke device K2 and filming viewer U2 with imaging means 60. The viewer data processing unit 700 synthesizes an audio signal corresponding to the voice of viewer U2 collected by microphone 40 and a video signal corresponding to the video of viewer U2 filmed by imaging means 60 to generate viewer data. The viewer data processing unit 700 transmits the generated viewer data to server device S.

The viewer data processing unit 700 continuously generates and transmits viewer data from the start time included in the distribution information D1 until the end time arrives (i.e., until the live show L1 ends). If viewer U2 stops watching the live show L1 midway through, the viewer data processing unit 700 stops generating and transmitting the viewer data.

Here, the viewer data processing unit 700 according to this embodiment stops transmitting the viewer data if the excitement does not satisfy a predetermined condition. Specifically, if the determination unit 600 determines that the excitement of the viewers during a certain predetermined period does not satisfy a predetermined condition, the viewer data processing unit 700 stops transmitting the viewer data to the server device S for a certain period following the certain predetermined period.

For example, suppose that the determination unit 600 determines that the level of excitement of viewer U2 during a period P1, which is 10 seconds after the start of the karaoke performance of song X1, does not satisfy a predetermined condition. In this case, the viewer data processing unit 700 stops transmitting viewer data of viewer U2 to the server device S during a period P2 following period P1. After period P2 has elapsed, the determination unit 600 re-determines the level of excitement of viewer U2 during a period P3 following period P2.

==Server Device==
5, the server device S includes a storage unit 20, a communication unit 30, and a control unit 40. Each component is connected to a bus B via an interface (not shown).

[Storage means/communication means]
The storage means 20 is a large-capacity storage device for storing various data, such as a hard disk drive, etc. The communication means 30 provides an interface for communicating with each karaoke device.

[Control Means]
The control means 40 performs various controls in the server device S. The control means 40 includes a CPU and a memory (neither of which are shown in the figure). The CPU realizes various functions by executing programs stored in the memory.

In this embodiment, the CPU executes the programs stored in the memory, causing the control means 40 to function as a reservation management unit 40a, a live data generation unit 40b, and a distribution processing unit 40c.

(Reservation Management Department)
The reservation management unit 40a manages reservations for live distribution and viewing.

The reservation management unit 40a accepts reservations for live streaming based on the device identifier of the karaoke device, the distribution identifier, and the distribution information corresponding to the distribution identifier transmitted from the live streaming reservation unit 100 of one karaoke device. The reservation management unit 40a manages reservations for live streaming by storing the received identifiers, etc. in the storage means 20.

On the other hand, when an instruction to view a live broadcast is received from the live viewing reservation unit 400 of another karaoke device, the reservation management unit 40a reads out the broadcast identifier and broadcast information of the live broadcast scheduled to be broadcast in the live broadcast system 1 from the storage means 20 and transmits them to the other karaoke device.

In addition, when the reservation management unit 40a receives a distribution identifier of a live performance selected by a viewer from the live performance viewing reservation unit 400 of another karaoke device, it manages viewing by linking the device identifier of the other karaoke device to the live performance stored in the storage means 20.

As described above, it is assumed that the live streaming reservation unit 100 links the device identifier ID_K1 of the karaoke device K1, the streaming identifier ID_D1 of the live performance L1, and the streaming information D1, and transmits them to the server device S via the communication means 10b. It is also assumed that the live viewing reservation units 400 of the karaoke devices K2 to K5 link the device identifiers ID_K2 to ID_K5 of the karaoke devices K2 to K5 to the streaming identifier ID_D1 of the live performance L1 selected by the viewers U2 to U5, respectively, and transmit them to the server device S via the communication means 10b.

In this case, the reservation management unit 40a creates a table as shown in FIG. 6 and stores it in the storage means 20. By referring to this table, the server device S can link the distribution data transmitted from the karaoke device K1 with the viewer data transmitted from the karaoke devices K2 to K5.

(Live Data Generation Division)
The live data generating unit 40b generates live data based on the distribution data received from one karaoke device. When the live data generating unit 40b receives viewer data from another karaoke device, the live data generating unit 40b generates live data by combining the viewer data with the distribution data.

For example, suppose that a live broadcast L1 performed by a broadcaster U1 is watched by the broadcaster U1 and five viewers U2 to U5.

In this case, the distribution data processing unit 200 of the karaoke device K1 transmits distribution data corresponding to the live performance L1 to the server device S at the start time included in the distribution information D1. Meanwhile, the viewer data processing units 700 of the karaoke devices K2 to K5 transmit viewer data for viewing the live performance L1 to the server device S.

The live data generation unit 40b generates live data by combining distribution data with viewer data.

The live data includes audio data based on the voice of the broadcaster (greeting voice, singing karaoke, etc.) and the voice of the viewers (cheering, applause, etc.), and video data based on the video of the broadcaster (singing karaoke, etc.) and the video of the viewers (clapping, etc.).

The live data may include layout data. The layout data indicates the layout when the video based on the live data is displayed on the karaoke device. For example, as shown in FIG. 7, the layout may display video M1 of distributor U1 in the upper half, and video M2 to video M5 of viewers U2 to U5 in the lower half.

Now, let us say that the determination unit 600 of the karaoke device K2 determines the level of excitement during period P1 and determines that the excitement of viewer U2 does not satisfy the predetermined condition. On the other hand, let us say that the excitement of viewers U3 to U5 during period P1 does satisfy the predetermined condition. In this case, the viewer data processing unit 700 of the karaoke device K2 stops transmitting the viewer data of viewer U2 during period P2, which follows period P1.

The live data generator 40b therefore generates live data that combines the distribution data of distributor U1 with only the viewer data of viewers U3 to U5 as live data corresponding to period P2. Figure 8 shows an example of the layout of Figure 7, where video M1 of distributor U1 is displayed in the upper half and videos M3 to M5 of viewers U3 to U5 are displayed in the lower half.

If the excitement level does not meet the predetermined conditions, no viewer data will be transmitted from other karaoke devices. In this case, the live data generating unit 40b generates live data based on the distribution data.

In addition, when adopting a layout that displays four viewers as in Figure 7, there may actually be five or more viewers watching. In this case, the live data generator 40b selects the videos of the four viewers to compose the live data. The video selection is performed randomly.

(Distribution Processing Unit)
The distribution processing unit 40c distributes the generated live data to the karaoke device and other karaoke devices.

The distribution processing unit 40c distributes the generated live data to each karaoke device based on the device identifier. For example, in the example table shown in FIG. 6, live L1 is linked to device identifiers ID_K1 to ID_K5 of karaoke devices K1 to K5, including karaoke device K1 that distributes live L1. Therefore, the distribution processing unit 40c distributes the live data to karaoke devices K1 to K5.

The distributor-side setting unit 300 of the karaoke device K1 causes an image based on the live data of the live show L1 to be displayed on the display screen of the display device 30, and causes audio based on the live data to be output from the speaker 20. Similarly, the viewer-side setting units 500 of the karaoke devices K2 to K5 cause an image based on the live data of the live show L1 to be displayed on the display screen of the display device 30, and causes audio based on the live data to be output from the speaker 20. In this case, the images are displayed on each karaoke device in a layout such as that shown in FIG. 7, for example.

As described above, on the other hand, it is assumed that the live data generating unit 40b generates live data for period P2 by combining the distribution data of distributor U1 with the viewer data of viewers U3 to U5. In this case, the distribution processing unit 40c transmits this live data to karaoke devices K1 to K5. That is, the distribution processing unit 40c also transmits live data to karaoke device K2, which did not transmit viewer data. In this case, the video is displayed on each karaoke device, for example, in a layout as shown in FIG. 8.

==About the operation of the live streaming system==
Next, a specific example of the operation of the live streaming system 1 in this embodiment will be described with reference to Fig. 9 and Fig. 10. Fig. 9 is a flowchart showing an example of a basic operation of the live streaming system 1. Fig. 10 is a flowchart showing an example of an excitement determination operation of the live streaming system 1. In these examples, one karaoke machine is a karaoke machine used by a distributor who performs live streaming, and the other karaoke machine is a karaoke machine used by a viewer who watches the live performance.

[Basic operation example]
When the start time of the live broadcast arrives, the broadcast data processing unit 200 generates broadcast data based on the broadcaster's voice picked up by the microphone 40 of one of the karaoke machines and the video of the broadcaster captured by the filming means 60 of one of the karaoke machines, and begins transmitting the generated broadcast data to the server device S (start of transmission of broadcast data; step 10).

In addition, when the start time of the live broadcast arrives, the viewer data processing unit 700 generates viewer data based on the viewer's voice collected by the microphone 40 of the other karaoke device and the video of the viewer captured by the imaging means 60 of the other karaoke device, and starts transmitting the generated viewer data to the server device S (start of viewer data transmission; step 11).

The live data generation unit 40b of the server device S starts generating live data by combining the distribution data transmitted in step 10 with the viewer data transmitted in step 11 (start of live data generation; step 12).

The distribution processing unit 40c of the server device S starts distributing the live data generated in step 12 to the first karaoke device and the other karaoke devices (start of live data distribution; step 13).

The distributor setting unit 300 sets the broadcaster so that the broadcaster can view the audio and video based on the live data broadcast in step 13. The broadcaster starts viewing the audio and video of the live performance he or she is performing on one of the karaoke devices (start viewing audio and video; step 14).

Similarly, the viewer-side setting unit 500 sets the viewer to be able to view audio and video based on the live data distributed in step 13. The viewer starts viewing the distributed live audio and video on another karaoke device (start viewing audio and video; step 15).

The live distribution system 1 repeatedly transmits distribution data to the server device S, transmits the generated viewer data to the server device S, generates live data, and distributes the generated live data to one karaoke device and another karaoke device until the end of the live distribution. The distributor and viewers can listen to the live audio and video being distributed on the karaoke device until the end of the live distribution.

[Example of operation for excitement determination]
After the start of live distribution, it is assumed that the distributor issues an instruction to start a karaoke performance of a song included in the distribution information via remote control device 50 of one of the karaoke devices. In this case, the distribution data processing unit 200 notifies the server device S of the start of the karaoke performance by transmitting a performance start signal (notifying the start of karaoke performance; step 20).

Upon receiving the performance start signal in step 20, the server device S transfers the notification to the other karaoke devices (transfer of notification; step 21).

The distributor sings karaoke along with the karaoke performance of the song. The distribution data processing unit 200 synthesizes an audio signal corresponding to the distributor's singing voice and the karaoke performance of the song collected by the microphone 40 with a video signal corresponding to the video of the distributor captured by the imaging means 60, to generate distribution data. The distribution data processing unit 200 starts transmitting the generated distribution data to the server device S (start transmission of distribution data; step 22).

After receiving the notification in step 21, the judgment unit 600 judges whether the excitement of the viewer satisfies a predetermined condition based on the voice of the viewer picked up by the microphone 40 of the other karaoke device and/or the image of the viewer captured by the imaging means 60 of the other karaoke device.

If the excitement satisfies the predetermined condition (Y in step 23), the viewer data processing unit 700 generates viewer data based on the viewer's voice and viewer's video, and starts transmitting the generated viewer data to the server device S (start of viewer data transmission; step 24).

On the other hand, if the excitement does not satisfy the predetermined condition (N in step 23), the viewer data processing unit 700 stops transmitting the viewer data to the server device S (stop transmitting viewer data; step 25).

The live data generating unit 40b of the server device S starts generating live data by combining the distribution data transmitted in step 21 with the viewer data transmitted in step 24 (start of live data generation; step 26). Note that if the excitement does not satisfy a predetermined condition, the transmission of the viewer data is stopped. In this case, the live data generating unit 40b generates live data based only on the distribution data transmitted in step 21.

The distribution processing unit 40c of the server device S starts distributing the live data generated in step 26 to the first karaoke device and the other karaoke devices (start of live data distribution; step 27).

The distributor setting unit 300 sets the broadcaster so that the broadcaster can view the audio and video based on the live data broadcast in step 27. The broadcaster starts viewing the audio and video of the live performance he or she is performing on one of the karaoke devices (start viewing audio and video; step 28).

Similarly, the viewer-side setting unit 500 sets the viewer to be able to view audio and video based on the live data distributed in step 27. The viewer starts viewing the distributed live audio and video on another karaoke device (start viewing audio and video; step 29).

Until the karaoke performance of a song is completed, the live distribution system 1 repeatedly transmits data to be distributed to the server device S, determines whether the excitement meets a predetermined condition, transmits or stops transmitting the generated viewer data to the server device S, generates live data, and distributes the generated live data to the karaoke device and other karaoke devices.

As is clear from the above, in the live distribution system 1 according to this embodiment, karaoke devices K1 to Kn are communicatively connected to a server device S. The karaoke device K1 has a distribution data processing unit 200 that generates distribution data based on the voice of a distributor U1 who wishes to live distribute karaoke singing, which is collected by the microphone 40 of the karaoke device K, and a video of the distributor U1, which is captured by the imaging means 60 of the karaoke device K1, and transmits the generated distribution data to the server device S, and a distributor side setting unit 300 that sets the voice and video so that the distributor U1 can view them based on the live data distributed from the server device S. The karaoke devices K2 to Kn each have a viewer setting unit 500 that sets the viewer to be able to view audio and video based on live data distributed from the server device S, a determination unit 600 that determines whether the viewer's excitement satisfies a predetermined condition based on the viewer's audio collected by the microphone 40 of the karaoke device K2 to Kn and/or the viewer's video captured by the imaging means 60 of the karaoke device K2 to Kn, and a viewer data processing unit 700 that generates viewer data based on the viewer's audio and video and transmits the generated viewer data to the server device S, and if the excitement does not satisfy the predetermined condition, stops transmitting the viewer data to the server device S. The server device S has a live data generating unit 40b that generates live data based on the distribution data received from the karaoke device K1, and when viewer data is received from the karaoke devices K2 to Kn, generates live data by combining the viewer data with the distribution data, and a distribution processing unit 40c that distributes the generated live data to the karaoke device K1 and the karaoke devices K2 to Kn.

With this kind of live streaming system 1, the viewer's voice and video can be viewed along with the voice and video of the broadcaster who is live streaming. This allows the broadcaster to feel as if they are actually singing karaoke in front of the audience. The audience can also feel as if they are actually watching the karaoke being sung. Furthermore, with the live streaming system 1 according to this embodiment, a sense of unity in the live streaming can be provided for the broadcaster and the audience by combining only the voice and video of the audience who are enjoying the performance with the broadcaster's voice and video. In other words, with the live streaming system 1 according to this embodiment, when live streaming is performed using a karaoke device, the broadcaster's voice and video can be combined with the viewer's voice and video for broadcast.

In addition, in the live streaming system 1 according to this embodiment, the determination unit 600 determines the level of excitement among viewers at each predetermined period, and if the viewer data processing unit 700 determines that the level of excitement among viewers during a certain predetermined period does not satisfy a predetermined condition, it suspends the transmission of viewer data to the server device S for a certain period following the certain predetermined period. With this kind of live streaming system 1, it is possible to more accurately reflect the state of excitement in live streaming.

The determination unit 600 in the live streaming system 1 according to this embodiment can determine the level of excitement of a viewer based on the sound pressure level of a specific frequency band detected from the viewer's voice and/or the movement of the viewer detected from the video of the viewer. With this kind of live streaming system 1, it is possible to more accurately determine the level of excitement.

<Modification>
In the above embodiment, the determining unit 600 may weight the excitement level based on the degree of agreement between the distributor's history information and the viewer's history information.

==Server Device==
11, the server device S according to this modification includes a storage unit 20, a communication unit 30, and a control unit 40. In this modification, the CPU executes a program stored in the memory, so that the control unit 40 functions as a reservation management unit 40a, a live data generation unit 40b, a distribution processing unit 40c, an acquisition unit 40d, and a calculation unit 40e.

(Acquisition Department)
The acquisition unit 40d acquires history information for each of the distributors and viewers.

The history information includes at least one of the attributes and singing history of the distributor or viewer. The attributes include the distributor's or viewer's hometown, age, gender, preferred music genre, etc. The singing history includes the song titles, singer names, and scoring results of karaoke performances performed by the distributor or viewer in the past. The storage means 20 of the server device S stores the history information in association with a dedicated identifier for identifying the distributor or viewer, such as a personal ID.

When using the karaoke device, the distributor and the viewer input their personal ID via the remote control device 50. The karaoke device transmits the input personal ID to the server device S. The acquisition unit 40d acquires the history information by reading the history information corresponding to the received personal IDs of the distributor and the viewer from the storage means 20.

(Calculation section)
The calculation unit 40e calculates a weighting value for determining the excitement level based on the degree of agreement between the acquired history information of the distributor and the history information of the viewer.

The weighting value can be calculated in various ways. For example, if the history information is a numerical value such as a score result or age, the calculation unit 40e can calculate the weighting value by finding the difference between the numerical values and referring to a pre-set table.

For example, assume that the weighting value table shown in FIG. 12 is stored in the storage means 20. Here, assume that the acquisition unit 40d acquires the age of distributor U1 (30 years old) and the age of viewer U2 (28 years old) as historical information. The calculation unit 40e obtains the difference (2 years) between the ages of distributor U1 and viewer U2, and obtains a weighting value of "1.2" by referring to the weighting value table shown in FIG. 12. The calculation unit 40e transmits the obtained weighting value to the karaoke device K2 used by viewer U2.

==Other Karaoke Equipment==
In this modification, the determining section 600 of the other karaoke apparatus determines the excitement level based on the weighting values.

For example, suppose that a weighting value of "1.2" is sent from calculation unit 40e to karaoke device K2. In this case, judgment unit 600 extracts a frequency band corresponding to a specific frequency band from the voice of viewer U2 picked up by microphone 40, and detects the average sound pressure level over a predetermined period of time. Then, judgment unit 600 obtains a correction level by multiplying the detected average sound pressure level by the weighting value, and judges whether the correction level is equal to or greater than a predetermined sound pressure level.

In this way, the server device S according to this modified example has an acquisition unit 40d that acquires history information including at least one of the attributes and singing history of the broadcaster or viewer for each broadcaster and viewer, and a calculation unit 40e that calculates a weighting value for determining excitement based on the degree of agreement between the acquired broadcaster history information and viewer history information. In addition, the determination unit 600 determines excitement based on the weighting value.

Distributors and viewers with similar history information are likely to be of similar generation or have similar hobbies and interests. In such cases, there is a high possibility that the viewer will become enthusiastic. Therefore, by weighting based on the degree of similarity in the history information, it is possible to more accurately determine the viewer's level of enthusiasm.

＜Other＞
In the above embodiment, an example has been described in which one karaoke device is used by one viewer. However, there may be cases in which one karaoke device is used by multiple viewers. In this case, the determination unit 600 may determine the excitement level based on the sense of unity among all the multiple viewers. For example, the determination unit 600 may determine the excitement level by analyzing the video of the viewers and detecting the number of viewers who are moving at a tempo that matches the tempo of the music. In this case, the determination unit 600 may determine that the excitement level is high when the number of detected viewers is equal to or greater than a predetermined number.

The above embodiments are presented as examples and do not limit the scope of the invention. The above configurations can be implemented in appropriate combinations, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The above embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and gist of the invention.

40a Reservation management section 40b Live data generation section 40c Distribution processing section 100 Live distribution reservation section 200 Distribution data processing section 300 Distributor side setting section 400 Live viewing reservation section 500 Viewer side setting section 600 Determination section 700 Viewer data processing section K1 to Kn Karaoke device S Server device

Claims (4)

A live distribution system in which a plurality of karaoke machines and a server device are communicatively connected,
One karaoke machine,
a data-for-delivery processing unit that generates data for delivery based on the voice of a distributor wishing to live-deliver karaoke singing, which is collected by a sound collection means of the first karaoke device, and a video of the distributor, which is photographed by a photographing means of the first karaoke device, and transmits the generated data for delivery to the server device;
a distributor-side setting unit that performs setting so that the distributor can view audio and video based on the live data distributed from the server device;
having
Another karaoke device different from the one karaoke device,
a viewer side setting unit that sets the live data distributed from the server device so that the viewer can view the audio and video;
a determination unit that determines whether or not the excitement of the viewer satisfies a predetermined condition based on the voice of the viewer collected by a sound collection unit of the other karaoke device and/or the image of the viewer captured by a capture unit of the other karaoke device;
a viewer data processing unit that generates viewer data based on the voices and images of the viewer and transmits the generated viewer data to the server device, and when the excitement does not satisfy a predetermined condition, stops transmitting the viewer data to the server device;
having
The server device includes:
a live data generating unit that generates live data based on the distribution data received from the one karaoke device, and when viewer data is received from the other karaoke device, generates live data by combining the distribution data with the viewer data;
a distribution processing unit that distributes the generated live data to the first karaoke device and the second karaoke device;
A live distribution system having the above configuration. The determination unit determines the level of excitement of viewers at each predetermined period of time,
The live streaming system described in claim 1, characterized in that if the viewer data processing unit determines that the viewer's enthusiasm during a certain specified period does not satisfy a specified condition, it stops transmitting viewer data to the server device for a certain period following the certain specified period. The live distribution system according to claim 1 or 2, characterized in that the determination unit determines the excitement of the viewer based on the sound pressure level of a specific frequency band detected from the viewer's voice and/or the movement of the viewer detected from the video of the viewer. The server device includes:
An acquisition unit that acquires history information including at least one of attributes and singing history of the distributor or the viewer for each of the distributor and the viewer;
A calculation unit that calculates a weighting value for determining the excitement level based on the degree of agreement between the acquired history information of the distributor and the history information of the viewer;
having
The live distribution system according to claim 1 , wherein the determination unit determines the excitement level based on the weighting value.

Images