JP7727233B2 - Information processing system, control program, and information processing device - Google Patents

Description

The present invention relates to an information processing system, a control program, and an information processing device.

There is a known mechanism for allowing viewers of streaming video content to participate in group chats on the topic of that video content. For example, Patent Document 1 discloses a client device that displays a standard message related to the content of the video content in response to a button selected by the user on a chat screen where avatars corresponding to the users participating in the group chat are arranged around the video content.

International Publication No. 2017/026170

The client device described in Patent Document 1 displays a standard message selected by the user in a speech bubble next to an avatar representing the user. This allows the user to see the reactions of other users watching the video content, but simply displaying a standard message selected by the user in a speech bubble is not very entertaining.

Patent Document 1 also discloses displaying speech bubbles in sizes that correspond to the frequency with which multiple users select standard messages. However, if standard messages from other users are displayed in large sizes, users watching video content may find this annoying.

The problem that this invention aims to solve is to improve entertainment value while minimizing annoyance in a system that displays viewer reactions on the display screen of the viewing target.

To solve the above problem, the present invention allows a first element based on the user's operation to be superimposed on the viewing target on a screen on which the viewing target is displayed, while preventing a second element based on the operation of another person from being superimposed on the viewing target.

An information processing system according to one embodiment of the present invention includes a first control means for displaying a first element based on a first operation by a first user viewing the viewing target on a first information processing device in a first range that at least partially overlaps with a basic range in which the viewing target is displayed on the first information processing device, and a second control means for displaying a second element based on a second operation by a second user viewing the viewing target on a second information processing device in a second range that does not overlap with the basic range.

This invention allows for a system that displays viewer reactions on the target display screen to reduce annoyance while improving entertainment value.

1 is a diagram illustrating the configuration of a video distribution system (embodiment). 1 is a block diagram showing the electrical configuration of a server (embodiment); 1 is a block diagram showing the electrical configuration of a mobile terminal (embodiment); 10A and 10B are diagrams illustrating the display mode of the screen of a mobile terminal (embodiment); 10 is a diagram showing a screen displaying an onomatopoeia image (embodiment); 10 is a diagram showing a screen displaying a second image (Example). 1 is a functional block diagram of a server for video distribution (embodiment); 1 is a functional block diagram relating to onomatopoeia display on a mobile terminal (embodiment); 10 is a flowchart showing the flow of an onomatopoeia display process executed on a mobile terminal (embodiment). 10 is a flowchart showing the flow of an onomatopoeia display process executed by the server (embodiment). 10 is a schematic diagram showing the configuration of a screen (another embodiment).

1. Embodiment
[1-1. Main Features]
The present invention relates to a mechanism for displaying viewer responses on a display screen of a viewing target. In order to reduce annoyance and improve entertainment value, each embodiment of the present invention displays a first element based on a first operation by a first user viewing the viewing target in a first range that at least partially overlaps with a basic area in which the viewing target is displayed, and displays a second element based on a second operation by a second user viewing the viewing target in a second range that does not overlap with the basic area.

The viewing object refers to video content that is at least visually perceived by the user. The video content may also include audio content that is audibly perceived by the user. The video content is distributed, for example, from a distribution device to an information processing device. The video content is preferably distributed, for example, by streaming distribution or progressive download distribution. The video content is, for example, a moving image. The viewing object is displayed in a basic area that is part of the screen of the information processing device.

A user (viewer) viewing a viewing object can add a predetermined element (additional element) to the viewing object by performing a predetermined operation (additional operation). The predetermined operation is, for example, a pointing operation on an input element (operation element). The input element is, for example, a button displayed on the screen. The pointing operation is, for example, a touch operation on a touch panel, a combination of a positioning operation of a pointer displayed on the screen, and a click operation. The predetermined element is, for example, an image representing a user's reaction to the viewing object. The user's reaction is, for example, the user's emotion. An image representing a user's emotion toward the viewing object is sometimes called an emotional image. An emotional image is, for example, an image that depicts a stylized onomatopoeia (also called onomatopoeic words, mimetic words, or the like) that represents the user's emotion. An image that depicts a stylized onomatopoeia is sometimes called an onomatopoeia image. Adding an element to a viewing object means associating the display timing of the element with the progress of the viewing object. When the display timing is associated with the progress, the display timing is specified at a specific position on the timeline. The elements added to the viewing target can be displayed on the screen at the associated display timing while the viewing target is in progress (i.e., while it is being played and displayed).

The first user is a user (the user himself/herself) who uses an information processing device. A predetermined operation (first operation) by the first user is accepted, for example, by the information processing device. The second user is another user (another person) who uses another information processing device. A predetermined operation (second operation) by the second user is accepted, for example, by the other information processing device. A predetermined element (first element) based on the first operation is displayed in a first range that at least partially overlaps the basic area. A predetermined element (second element) based on the second operation is displayed in a second range that does not overlap the basic area. In addition to a portion (second portion) that overlaps the basic area, the first range may also have a portion (first portion) that does not overlap the basic area. The first range may further have a portion (third portion) that overlaps the second range.

[1-2. Supplementary notes]
The present invention includes at least the following embodiments [A] to [D].

[A] An information processing system comprising: a first control means for displaying a first element based on a first operation by a first user viewing the viewing target on a first information processing device in a first range that at least partially overlaps with a basic range in which the viewing target is displayed on the first information processing device; and a second control means for displaying a second element based on a second operation by a second user viewing the viewing target on a second information processing device in a second range that does not overlap with the basic range.

[B] A control program that causes a computer of an information processing device included in an information processing system including multiple information processing devices used by multiple users viewing a viewing target to realize a first control function that causes a first element based on a first operation by a user viewing the viewing target to be displayed in a first range that at least partially overlaps with a basic area in which the viewing target is displayed, and a second control function that causes a second element based on a second operation by another user viewing the viewing target on another information processing device to be displayed in a second range that does not overlap with the basic area. Note that the control program may be non-temporarily recorded on a recording medium.

[C] An information processing device comprising: a receiving means for receiving a first operation by a first user viewing a viewing target; a first control means for displaying a first element based on the first operation received by the receiving means in a first range that at least partially overlaps with a basic area in which the viewing target is displayed; an acquisition means for acquiring information on a second operation by a second user viewing the viewing target from a management means that manages information on the second operation, and a second control means for displaying a second element based on the second operation acquired by the acquisition means in a second range that does not overlap with the basic area.

[D] A control method for an information processing device, comprising: a reception step of receiving a first operation by a first user viewing a viewing target; a first control step of displaying a first element based on the first operation received in the reception step in a first range that at least partially overlaps with a basic area in which the viewing target is displayed; an acquisition step of acquiring information on a second operation by a second user viewing the viewing target from a management means that manages information on the second operation, and a second control step of displaying a second element based on the second operation acquired in the acquisition step in a second range that does not overlap with the basic area.

The following technical limitations may be added to the "information processing system" in [A] above. Similar technical limitations may also be added to the "control program" in [B] above, the "information processing device" in [C] above, and the "control method for an information processing device" in [D] above.

(1) The first information processing device further includes a receiving means for receiving the first operation via an operation element that is positioned so as not to overlap the basic area. As a result, the element based on the operation received by the information processing device becomes the first element.

(2) The first control means displays the first element in the first range, which has a first portion that does not overlap either the basic region or the second range, and a second portion that overlaps the basic region, while changing the display mode from the first portion to the second portion. The display mode can be, for example, size, shape, position, or number of components. The first element, which is displayed while changing its display mode from the first portion to the second portion, changes from a state where it is not overlapped with the viewing target to a state where it is overlapped with the viewing target. This further improves entertainment value.

(3) The first control means displays the first element in the first range, which further has a third portion superimposed on the second range, while changing the display mode from the first portion, through the second portion, to the third portion. The display mode may be, for example, size, shape, position, or number of components. The first element, which is displayed while changing the display mode from the first portion, through the second portion, to the third portion, changes from a state where it is not superimposed on the viewing target to a state where it is superimposed on the viewing target, and also changes from a state where it cannot be superimposed on the second element to a state where it can be superimposed on the second element. This further enhances entertainment value.

(4) The first control means displays the first element in the first range while changing the display mode so that the first element extends from a point near the operating element located in the first portion. This further enhances entertainment value.

2. Examples
An embodiment of the present invention will be described below with reference to the drawings. This embodiment relates to a mechanism for displaying an image (image showing a reaction) representing the emotion of a user (viewer) on a display screen of a video (viewing target). In this embodiment, the information processing device will be described as a mobile terminal, a so-called smartphone. Note that the mobile terminal is not limited to a smartphone, but may also be a tablet terminal, a laptop computer, or the like. The information processing device may also be a desktop personal computer. Each program according to the embodiment is stored in the information processing device, but is not limited thereto, and may also be stored in a storage medium such as an optical disk such as a CD-ROM (Compact Disk Read Only Memory) or a portable semiconductor memory such as a USB (Universal Serial Bus) flash memory.

[2-1. Features]
In this embodiment, in order to reduce annoyance while improving entertainment value, a first image (additional element, first element) based on an additional operation (first operation) by a first user (user) watching a video is displayed in a range (first range) that partially overlaps with the streaming video display area (basic area), and is extended (displayed while changing the display mode) from the first image display area, starting from the vicinity of the input element (operation element) placed in the first image display area (first part), through the streaming video display area (second part), and toward the second image display area (third part), and a second image (additional element, second element) based on an additional operation (second operation) by a second user (other person) watching a video is displayed in a range (second range) that does not overlap with the streaming video display area.

Furthermore, in this embodiment, in order to allow the viewer of a video to efficiently specify elements to be added to the video, while the video and input elements (operation elements) are displayed on the screen, a touch operation (addition operation) is accepted to display on the screen a first image (additional element) that represents the user's (own) feelings (reaction) toward the video, and when the touch operation is accepted, the image is displayed on the screen in an expanded form (displayed while changing its display mode) the longer the press and hold time (duration of the addition operation) is.

[2-2. Video distribution system]
The mobile terminal according to the embodiment is included in a video distribution system, and videos are distributed from a server.

[2-2-1. Overall configuration of video distribution system]
1 is a schematic diagram of a video distribution system 1 according to an embodiment. The video distribution system 1 includes a communication line 2, a plurality of mobile terminals 3, and a server 4.

Communication line 2 forms a computer network and is, for example, a wide-area communication line provided by an electric utility company.

The mobile terminal 3 is, for example, an information processing terminal such as a smartphone, tablet terminal, or laptop computer. The mobile terminal 3 is equipped with a touch panel display 3a for displaying images, a speaker 3b for outputting sound, a microphone 3c for inputting sound, and a camera 3d for capturing images of a subject. In order to display videos distributed from the server 4, for example, dedicated application software (hereinafter referred to as a "video viewing app") is installed on the mobile terminal 3.

The server 4 is an information processing device that distributes content to multiple mobile terminals 3 via the communication line 2. The content in the embodiment is, for example, a video. The video may be, for example, an animation or drama that has a story and progresses along the story, or a music video, a live music video, or a variety of sports videos, but is not limited to this. The video in the embodiment is, for example, an animation. Furthermore, the content distributed from the server is not limited to a video, and may be other services, such as a collection of consecutive still images, as long as the content is viewable by the user.

Furthermore, viewing here means that the user perceives the content using at least one of their senses of sight and hearing, and in addition to the above, the content may also be, for example, video without sound, or sound without images, etc.

As an example, a plurality of mobile terminals 3 to which videos are distributed are registered in the server 4 as a group 5 (hereinafter referred to as "group registration"). The server 4 distributes videos simultaneously to the group-registered mobile terminals 3. That is, the group-registered mobile terminals 3 display the videos distributed from the server 4 at the same time, in other words, synchronized, on their touch panel displays 3a. This allows each user of the group-registered mobile terminals 3 to watch the videos simultaneously.

The video may be distributed at a predetermined date and time, or at any timing for each group 5. When video is distributed at any timing, for example, the host mobile terminal 3 requests the server 4 to distribute the video, and the server 4, upon receiving the distribution request, distributes the video to the mobile terminals 3 that make up the group 5.

In Figure 1, group 5 is made up of four mobile terminals 3, but this is just an example, and the number of mobile terminals 3 in group 5 is not limited as long as group 5 is made up of multiple mobile terminals 3. Furthermore, the types of mobile terminals 3 that make up group 5 are also not limited, and a mixture of smartphones, tablet terminals, laptops, etc. may be present.

[2-2-2. Server configuration]
FIG. 2 is a block diagram showing the electrical configuration of the server 4 according to the embodiment.

The server 4 according to the embodiment includes a CPU (Central Processing Unit) 20, which is the main control unit that controls the overall operation of the server 4; a ROM (Read Only Memory) 22 in which various programs and data are pre-stored; a RAM (Random Access Memory) 24, which is used as a work area when the CPU 20 executes various programs; and a HDD (Hard Disk Drive) 26, which serves as storage means for storing various programs and data.

The HDD 26 stores information about the registered group 5, videos (video data) to be distributed to the mobile terminal 3, and various other information. Note that the storage means is not limited to the HDD 26, and may be other storage media, such as semiconductor memory such as flash memory.

Furthermore, the server 4 is equipped with an operation input unit 28, which is composed of a keyboard, mouse, etc. and accepts input for various operations, a monitor 30, such as a liquid crystal display device, that displays various images, and an external interface 32, which is connected to other information processing devices such as a mobile terminal 3 via a communication line 2 and sends and receives various data to and from other information processing devices, etc.

The CPU 20, ROM 22, RAM 24, HDD 26, operation input unit 28, monitor 30, and external interface 32 are electrically connected to one another via a system bus 34. Therefore, the CPU 20 can access the ROM 22, RAM 24, and HDD 26, grasp the operation status of the operation input unit 28, display images on the monitor 30, and send and receive various data to and from other information processing devices via the external interface 32.

[2-2-3. Electrical configuration of mobile terminal]
Fig. 3 is a functional block diagram showing the electrical configuration of the mobile terminal 3. In addition to the configuration shown in Fig. 1, the mobile terminal 3 includes a main control unit 40, a main memory unit 42, an auxiliary memory unit 44, a communication unit 46, and operation buttons 48.

The main control unit 40 is, for example, a CPU, microprocessor, DSP (Digital Signal Processor), etc., and controls the overall operation of the mobile terminal 3.

The main memory unit 42 is composed of, for example, RAM or DRAM (Dynamic Random Access Memory), and is used as a work area when the main control unit 40 executes processes based on various programs.

The auxiliary memory unit 44 is a non-volatile memory such as a flash memory, and stores various data such as images, as well as programs used in processing by the main control unit 40. The programs stored in the auxiliary memory unit 44 include, for example, an operating system (OS) for implementing the basic functions of the mobile terminal 3, drivers for controlling various hardware, and programs for implementing email, web browsing, and various other functions. The auxiliary memory unit 44 also pre-stores application software such as a video viewing app for playing videos distributed from the server 4 and displaying them on the touch panel display 3a.

The communication unit 46 is, for example, a NIC (Network Interface Controller) and has the function of connecting to the communication line 2. Note that instead of or in addition to the NIC, the communication unit 46 may have the function of connecting to a wireless LAN (Local Area Network), the function of connecting to a wireless WAN (Wide Area Network), and the function of enabling short-range wireless communication such as Bluetooth (registered trademark), infrared communication, etc.

The operation buttons 48 are provided on the side of the mobile terminal 3 and include a power button for starting or stopping the mobile terminal 3 and a button for adjusting the volume of the sound output by the speaker 3b.

The main control unit 40, main memory unit 42, auxiliary memory unit 44, communication unit 46, touch panel display 3a, speaker 3b, microphone 3c, camera 3d, and operation buttons 48 are electrically connected to one another via a system bus 49. Therefore, the main control unit 40 can access the main memory unit 42 and auxiliary memory unit 44, display images on the touch panel display 3a, grasp the user's operation status on the touch panel display 3a and operation buttons 48, input sound to the microphone 3c, output sound from the speaker 3b, control the camera 3d, and access various communication networks and other information processing devices via the communication unit 46.

[2-3. Onomatopoeia display]
The mobile terminal 3 according to the embodiment is capable of displaying images (onomatopoeia images, the details of which will be described later) in response to user operations while displaying videos distributed from the server 4 on the touch panel display 3a. In the following description, the function of displaying onomatopoeia images on the touch panel display 3a is referred to as an onomatopoeia display function, and the touch panel display 3a is also referred to as a screen 3a.

The onomatopoeia display function is, for example, one of the functions of the video viewing app according to the embodiment. Note that, for example, onomatopoeia display is not possible for all videos displayed on screen 3a using the video viewing app; videos for which onomatopoeia display is possible are predetermined.

[2-3-1. Screen configuration]
4 to 6 show display modes of a screen 3a of a mobile terminal 3 on which a video viewing application according to the embodiment is running. The screen 3a has a distribution video display area 50A, a first image display area 50B, and a second image display area 50C.

The distributed video display area 50A is an area for displaying videos distributed from the server 4. The first image display area 50B is an area for displaying a first image 51 (see FIG. 5) in response to a user's operation. The second image display area 50C is an area for displaying a second image 52 (see FIG. 6) based on the first image 51 displayed on another mobile device 3 in response to a user's operation. As will be described in detail below, the second image 52 is a reduced image of the first image 51 displayed in the first image display area 50B of the other mobile device 3. In addition, the first image display area 50B and the second image display area 50C in this embodiment are, as an example, positioned opposite each other with the distributed video display area 50A in between.

Here, the first image 51 is an image including an emotional image that shows the emotion of the user watching the video. The emotional image in this embodiment is, for example, an image in which an onomatopoeic word (also called a mimetic sound word or an onomatopoeic word) that shows the emotion of the user is illustrated, in other words, an onomatopoeic image. Examples of onomatopoeic images include "Awesome," "Awesome," and "Bravo." The stylized image is, for example, a decorative drawing of letters that represent the onomatopoeic word.

In addition to onomatopoeia images, the first image 51 may also include various other images, such as character images showing characters appearing in the video, dialogue images showing the characters' lines, and text entered by the user themselves.

The first image 51 may also differ depending on the scene in the video. For example, it may be possible to display an onomatopoeia image or character image that the user is likely to select in response to a happy scene, a sad scene, or a scene involving a confrontation between characters.

Furthermore, the first images 51 that can be displayed may differ depending on the user. For example, the types of first images 51 that can be displayed may increase by obtaining a specific item through a service such as a game related to the distributed video. Furthermore, the types of first images 51 that can be displayed may increase, for example, upon payment of a predetermined fee (for example, monetary value such as currency, virtual currency, or points).

In the embodiment, the first image display area 50B displays multiple display buttons 53A-53D, which are operation units operated by the user and each correspond to a different first image 51. This allows the user to display the first image 51 in the first image display area 50B by operating the display buttons 53A-53D while watching a video. Note that, for example, operating the display buttons 53A-53D means touching (pressing) the display buttons 53A-53D with an indicator. For example, the indicator in the embodiment is the user's finger. In the following description, when the display buttons 53A-53D are not to be distinguished from one another, the alphabetical notation at the end of the name will be omitted.

In the examples of Figures 4 to 6, four display buttons 53A to 53D are displayed in a row horizontally at the bottom of the first image display area 50B. Display button 53A corresponds to an onomatopoeia image that is a stylized version of "suge," display button 53B corresponds to an onomatopoeia image that is a stylized version of "yabai," and display button 53C corresponds to an onomatopoeia image that is a stylized version of "bravo." Furthermore, display button 53D corresponds to a character image. The display buttons 53 are designed to allow the user to intuitively recognize the corresponding onomatopoeia image, character image, etc.

In this way, by displaying multiple display buttons 53 in the first image display area 50B, the user can easily display on the screen 3a the first image 51 that best matches their reaction (emotion) to the video. Furthermore, by including an onomatopoeia image that indicates the user's emotion in the first image 51, the user can display on the screen 3a their own emotions after watching the video.

In the examples of Figures 4 to 6, four types of display buttons 53 are displayed in the first image display area 50B, but this is just one example, and the number and types of display buttons 53 are not limited to this. Furthermore, the display buttons 53 may be switched as appropriate so that different first images 51 can be displayed depending on the scene in the video, etc.

The arrangement of the display buttons 53 is also one example, and may be displayed together on the right or left side of the first image display area 50B, or displayed in two horizontal rows at the bottom of the first image display area 50B. Furthermore, for example, when the user swipes left or right at the bottom of the screen 3a, new display buttons 53 for displaying different types of first images 51 may be displayed.

[2-3-2. Display of first image]
The display mode of the first image 51 will be described in detail with reference to Fig. 5. In the example of Fig. 5, the user operates the display button 53C, and an onomatopoeia image in which "Bravo" is stylized is displayed on the screen 3a.

The onomatopoeia display function according to the embodiment accepts a user operation to display an onomatopoeia image (first image 51) on screen 3a while a video is displayed on screen 3a. In other words, while watching a video, the user operates display button 53 to display on screen 3a an onomatopoeia image that corresponds to the emotion felt while watching the video.

The onomatopoeia image according to the embodiment is displayed on the screen 3a displaying the video, changing depending on how the user operates the display button 53. As an example, the onomatopoeia image according to the embodiment is displayed so as to extend from the first image display area 50B into the streaming video display area 50A depending on how the user operates the display button 53.

In this way, the user can change the onomatopoeia image displayed on the screen 3a by operating the mobile terminal 3 while watching a video. Therefore, the user can easily display the emotions they felt while watching the video on the screen while watching the video.

In addition, in the example of Figure 5, the onomatopoeia image is displayed beyond the first image display area 50B and superimposed on the distributed video display area 50A. In this way, the onomatopoeia display function according to the embodiment superimposes the onomatopoeia image beyond the first image display area 50B onto the distributed video display area 50A in accordance with the user's operation mode, making it easier for the user to empathize with the video they are watching, and making the video more entertaining to watch.

In Figure 5, the onomatopoeia image extends beyond the streaming video display area 50A and is also superimposed on the second image display area 50C. However, this is just one example, and the onomatopoeia image may not be superimposed on the second image display area 50C.

Next, we will explain in detail the onomatopoeia images that change depending on how the display button 53 is operated.

In the embodiment, the display mode of the first image 51 changes depending on the duration of the user's operation of the display button 53. The duration of the operation of the display button 53 is, for example, the time that the display button 53 is continuously pressed (hereinafter referred to as "long press time").

When the user presses the display button 53, the onomatopoeia display function according to this embodiment starts displaying the onomatopoeia image starting from a position near the position of the display button 53. Displaying the onomatopoeia image starting from a position near the position of the display button 53 means, for example, that the onomatopoeia image is displayed so that the bottom edge of the onomatopoeia image is approximately aligned with the position of the display button 53.

The onomatopoeia image then extends (hereinafter referred to as "extended display") from the first image display area 50B toward the streaming video display area 50A depending on the length of time the display button 53 is pressed.

Extended display, for example, displays the onomatopoeia image in a long, elongated or enlarged form. Note that extended display may also extend the onomatopoeia image while changing its form, such as the number of characters or the type of characters. For example, if the onomatopoeia image is "bravo," it may be extended so that an "a" is added between "bra" and "bo" depending on the long press time, or it may be extended so that an "o" is added between "bravo" and "-" depending on the long press time.

In other words, if the user feels a stronger emotion toward the video they are watching, they continue to press the display button 53 to display the onomatopoeia image that corresponds to that emotion. As a result, the onomatopoeia image is displayed extended from the first image display area 50B toward the streaming video display area 50A, and the onomatopoeia image is displayed on the screen 3a so that it stands out more.

In this way, the onomatopoeia image included in the first image 51 changes depending on the user's operation, allowing the user to display their own reaction (emotions) to the content on the screen 3a in a more entertaining manner.

In addition, the first image 51 is displayed so that it extends from the first image display area 50B toward the streaming video display area 50A depending on the user's operation, allowing the user to display their own reaction (emotions) to the content on the screen 3a in a more entertaining manner.

When the user finishes operating the display button 53 (long press operation), the display of the onomatopoeia image on the screen 3a stops. The display of the onomatopoeia image may stop, for example, by the displayed onomatopoeia image instantly disappearing from the screen 3a, or by the expanded onomatopoeia image shrinking toward the display button 53 and disappearing from the screen 3a.

In addition, in the example of Figure 5, the area around the display button 53C being operated by the user to display the onomatopoeia image is highlighted. This allows the user to easily recognize the display button 53C they are operating, and makes the onomatopoeia display more entertaining.

In addition, in the embodiment, the first image 51 other than the onomatopoeia image, such as a character image, is displayed on the screen 3a without changing, but this is not limited to this. The first image 51 other than the onomatopoeia image may also be displayed on the screen 3a displaying the video while changing depending on the manner in which the user operates the display button 53.

[2-3-3. Other display examples of the first image]
The operation for displaying the first image 51 and the display mode of the first image 51 are not limited to the above-described modes, and may be other modes exemplified below.

The display mode of the onomatopoeia image may change not only in response to a long press operation, but also, for example, in response to a gesture operation made by the user on screen 3a. This allows the user to easily display the emotions felt while watching the video on screen 3a while watching the video.

A gesture operation for displaying onomatopoeia is, for example, an operation in which the user presses the display button 53 and flicks it in the desired direction, a so-called flick operation. When a flick operation is performed, for example, the onomatopoeia image is displayed in the direction of the flick operation, starting from a position near the position of the display button 53. That is, if the user presses the display button 53 and flicks it to the upper right, the onomatopoeia image is displayed in the upper right direction, starting from the position of the display button 53. Furthermore, the manner in which the onomatopoeia image is elongated may differ depending on the speed of the flick operation. For example, the faster the flick speed, the more the onomatopoeia image is elongated, or the faster the flick speed, the faster the onomatopoeia image is elongated.

Furthermore, if the user who pressed the display button 53 slides their finger without lifting it from the screen 3a, the onomatopoeia image may be elongated from the position of the display button 53 as the user's finger moves. Furthermore, if the display button 53 is pressed repeatedly, such as by double tapping, the onomatopoeia image may be elongated according to the number of consecutive presses.

Furthermore, changes in the display mode of the onomatopoeia image are not limited to elongated display. For example, when the user performs a flick operation, the onomatopoeia image may be displayed starting from a position near the position of the display button 53 and moving in the direction of the flick operation without changing its form. In this case, the movement speed of the onomatopoeia image may change depending on the speed of the flick operation; for example, the faster the flick operation, the faster the movement speed of the onomatopoeia image.

Also, the onomatopoeia image may be displayed largest at the beginning and gradually shrink depending on the length of time the image is pressed.

Furthermore, the onomatopoeia image does not necessarily have to be displayed starting from the position of the display button 53. For example, when the display button 53 is pressed, the onomatopoeia image may be displayed in the center of the screen 3a (distribution video display area 50A), and may be extended vertically depending on the length of time the button is pressed.

Also, instead of the onomatopoeia image being displayed when the display button 53 is pressed with a finger, the onomatopoeia image may be displayed when the finger is released from the display button 53. In this case, too, the display mode of the onomatopoeia image changes depending on, for example, how long the display button 53 is pressed and held.

Furthermore, the size or shape of the onomatopoeia image may not change regardless of the operation mode of the display button 53. For example, when the user operates the display button 53, an onomatopoeia image of a size that superimposes on the first image display area 50B, the streaming video display area 50A, and the second image display area 50C may be displayed for a predetermined time.

[2-3-4. Display of second image]
The display mode of the second image 52 will be described in detail with reference to Fig. 6. As described above, the second image 52 is a reduced image of the first image 51 displayed on the other mobile terminal 3 by an operation of the other user.

The onomatopoeia display function according to the embodiment displays the second image 52 in the second image display area 50C with almost no time delay from the timing at which another user displays the first image 51 on another mobile device 3. This allows the user to know the reactions of other users who are watching the same video while watching the video.

Furthermore, if the second image 52 is displayed superimposed on the streaming video display area 50A like the first image 51, the user may find the second image 52 to be a nuisance, interfering with their viewing of the video. Therefore, the second image 52 is not displayed superimposed on the streaming video display area 50A beyond the second image display area 50C, regardless of the operation mode of other users. This allows the user to check other users' reactions to the video they are watching without feeling annoyed.

As mentioned above, the first image 51 can also be superimposed on the streaming video display area 50A, but this is done at the user's discretion, so the user will not find the superimposition of the first image 51 on the streaming video display area 50A bothersome.

In addition, the screen 3a is configured so that the first image display area 50B and the second image display area 50C are different areas, so that the first image 51 displayed by the user and the second image 52 displayed by another user are not displayed together. This allows the user to easily distinguish between the first image 51 displayed by the user and the second image 52 displayed by another user.

Furthermore, in this embodiment, other users are, as an example, other users who are registered in the same group as the user. In other words, if a user is watching a video without being registered in a group, the second image 52 will not be displayed in the second image display area 50C. However, this is not limited to this, and second images 52 generated by operations of all other users watching the video may be displayed in the second image display area 50C, regardless of whether they are registered in a group.

Furthermore, the second image display area 50C according to the embodiment displays multiple second images 52 created by operations by multiple other users. As an example, the second image display area 50C according to the embodiment does not specify the position at which the second image 52 is displayed for each user. That is, the second image display area 50C displays multiple second images 52 created by operations by multiple users in a mixed manner at random positions. Furthermore, the second images 52 may be displayed while moving up, down, left, or right within the second image display area 50C. However, this is not limited to this, and the position at which the second image 52 is displayed within the second image display area 50C may be specified for each user.

Furthermore, the second image 52 is displayed adjacent to an identification image (hereinafter referred to as a "user icon") 54 that indicates the other user who displayed the first image 51 designated as the second image 52 on their own mobile device 3. This allows the user to recognize which user created the second image 52 displayed in the second image display area 50C. Note that in the example of Figure 6, the user icon 54 is a circle with a single alphabet character displayed in the center, but this is just one example; the user icon 54 may be an image set for each user or the user's initials; the design of the user icon 54 is not limited.

In addition, the second image 52 shown in Figure 6 displays the onomatopoeia image horizontally, but this is set in accordance with the size (aspect ratio) of the second image display area 50C, and the second image 52 may also display the onomatopoeia image vertically.

Furthermore, although the second image 52 is not superimposed on the streaming video display area 50A, it may be displayed while changing (e.g., expanding) in the second image display area 50C in response to the operation mode of another user. Note that the second image 52 may be superimposed from the second image display area 50C onto the first image display area 50B, as long as it is not superimposed on the streaming video display area 50A.

[2-4. Functional Blocks]
Next, the onomatopoeia display will be described using a functional block diagram. Note that the function of the server 4 relating to the onomatopoeia display is included in the video distribution function.

[2-4-1. Functional block related to server video distribution function]
FIG. 7 is a functional block diagram of the video distribution function of the server 4 according to the embodiment.

The CPU 20 of the server 4 includes a group registration unit 60, a video distribution unit 61, a first image display information reception control unit 62, a second image display information generation unit 63, and a second image display information transmission control unit 64. Each function of the CPU 20 is realized by a program stored in the HDD 26.

The group registration unit 60 accepts registration of a group 5 consisting of multiple mobile devices 3. Note that, as an example, registration of a group 5 is performed by registering a user ID.

The video distribution unit 61 distributes videos to the mobile terminal 3 in response to a request from the mobile terminal 3.

The first image display information reception control unit 62 accepts first image display information received from the mobile terminal 3 via the external interface 32, associates the first image display information with the corresponding video, and stores it in the HDD 26.

The first image display information is information relating to the first image 51 displayed on the screen 3a of the mobile terminal 3, and is transmitted from the mobile terminal 3 to the server 4 at the time the first image 51 is displayed on the mobile terminal 3. The first image display information includes, for example, the type of the displayed first image 51, the user ID indicating the user who displayed the first image 51, the type of operation performed by the user to display the first image 51, and the elapsed time from the start of the video when the first image 51 was displayed.

The second image display information generation unit 63 generates second image display information based on the received first image display information, associates the second image display information with the corresponding video, and stores it in the HDD 26.

The second image display information is information about the second image 52 to be displayed in the second image display area 50C, and includes the type of second image 52, the user ID of the user who displayed the first image 51 that is the basis for the second image 52 on their own mobile device 3, etc.

The second image display information transmission control unit 64 controls the external interface 32 to transmit the second image display information to the mobile terminal 3. The destination of the second image display information is another mobile terminal 3 different from the mobile terminal 3 that transmitted the first image display information.

[2-4-2. Functional block related to onomatopoeia display on mobile devices]
FIG. 8 is a functional block diagram relating to onomatopoeia display of the mobile terminal 3 according to the embodiment.

The main control unit 40 provided in the mobile terminal 3 includes an image display control unit 70, a display area setting unit 71, a first image display operation reception unit 72, an operation status determination unit 73, a first image display information generation unit 74, a first image display information transmission control unit 75, and a second image display information reception control unit 76. Each function provided in the main control unit 40 is realized by a program stored in the auxiliary memory unit 44.

The image display control unit 70 controls the touch panel display 3a to display an image based on the image data. The image display control unit 70 according to the embodiment displays a video distributed from the server 4 in the distributed video display area 50A, displays a first image 51 in the first image display area 50B in response to user operation, and displays a second image 52 based on the first image 51 displayed on another mobile device 3 by another user in the second image display area 50C. Furthermore, the image display control unit 70 according to the embodiment displays the first image 51 beyond the first image display area 50B, superimposed on the distributed video display area 50A, in response to the user's operation mode.

The display area setting unit 71 sets the position and size (length and width) of the distribution video display area 50A, first image display area 50B, and second image display area 50C as the display mode on the screen 3a when a video is distributed from the server 4. Note that these positions and sizes are predetermined, but may also be set arbitrarily by the user.

The first image display operation receiving unit 72 receives a user's operation of the display buttons 53 while the screen 3a is displaying a video. That is, the first image display operation receiving unit 72 recognizes the display button 53 operated by the user from among the multiple display buttons 53, and causes the first image 51 corresponding to that display button 53 to be displayed on the screen 3a via the image display control unit 70. The first image display operation receiving unit 72 displays the first image 51 on the screen 3a so that the display mode of the first image 51 changes depending on the determination result by the operation state determination unit 73.

The operation state determination unit 73 determines the operation state of the display button 53 by the user. The operation state here refers to an operation mode including, for example, the length of time the display button 53 is pressed and held.

The first image display information generation unit 74 generates first image display information corresponding to the first image 51 displayed on the screen 3a in response to a user operation.

The first image display information transmission control unit 75 controls the communication unit 46 to transmit the first image display information to the server 4.

The second image display information reception control unit 76 outputs the second image display information received from the server 4 via the communication unit 46 to the image display control unit 70.

[2-5. Processing flow]
Next, the flow of the onomatopoeia display process in the mobile terminal 3 and the server 4 will be described using a flowchart.

[2-5-1. Flow for displaying onomatopoeia on a mobile device]
9 is a flowchart showing the flow of an onomatopoeia display process according to this embodiment, which is executed by the main control unit 40 of the mobile terminal 3. A program for executing the onomatopoeia display process is pre-stored in a predetermined area of the auxiliary storage unit 44. As an example, the onomatopoeia display process is started when the mobile terminal 3 starts playing a video distributed from the server 4, and is ended when the playback of the video ends.

In step S100, the occurrence of a trigger is detected. In this embodiment, the trigger is either a first trigger (acceptance of user operation of the display button 53) or a second trigger (reception of second image display information). The mobile terminal 3 in this embodiment updates the display content on the screen each time the occurrence of a trigger is detected. The first trigger and the second trigger can occur in parallel. Therefore, while processing is being executed in response to the occurrence of one trigger, processing is sometimes executed in response to the occurrence of the other trigger. Furthermore, multiple second triggers can occur in parallel. Therefore, while processing is being executed in response to one second trigger, processing is sometimes executed in response to another second trigger.

In step S102, if the detected trigger is the first trigger, proceed to step S106; if the detected trigger is the second trigger, proceed to step S104.

In step S104, the image display control unit 70 displays the second image 52 indicated by the second image display information in the second image display area 50C, and then the process ends.

In step S106, the image display control unit 70 displays the first image 51 corresponding to the display button 53 operated by the user in the first image display area 50B. Note that the first image 51 may be stored in advance on the mobile terminal 3, or may be downloaded from the server 4 each time the display button 53 is operated by the user.

In step S108, the first image display information generation unit 74 generates first image display information corresponding to the first image 51 displayed in the first image display area 50B.

In step S110, the first image display information transmission control unit 75 controls the communication unit 46 to transmit the first image display information to the server 4.

In step S112, the operation status determination unit 73 determines whether the user has finished operating the display button 53. If the determination is affirmative, the process proceeds to step S116; if the determination is negative, the process proceeds to step S114.

In step S114, the image display control unit 70 displays the first image 51 (onomatopoeia image) in an elongated format until the user finishes operating the display button 53, and then the process returns to step S112.

In step S116, the image display control unit 70 ends the display of the first image 51 and ends the process.

In the flowchart shown in FIG. 9, first image display information is generated and transmitted to the server 4 along with the display of the first image 51. However, this is not limited to this; the first image display information may be transmitted to the server 4 after the display of the first image 51 has ended. Furthermore, the first image display information transmission control unit 75 may transmit first image display information indicating the start of display of the first image 51 to the server 4 when the display of the first image 51 begins, and may transmit first image display information indicating the end of display of the second image 52 to the server 4 when the display of the first image 51 has ended. This allows the server 4 to recognize that the first image 51 (onomatopoeia image) is displayed in an elongated format during the interval between receiving the first image display information indicating the start of display of the first image 51 and the first image display information indicating the end of display.

[2-5-2. Flow related to onomatopoeia display on the server]
10 is a flowchart showing the flow of an onomatopoeia display process according to this embodiment, which is executed by the CPU 20 of the server 4. A program for executing the onomatopoeia display process is pre-stored in a predetermined area of the HDD 26. As an example, the onomatopoeia display process starts when the server 4 is distributing a video to the mobile terminal 3, and ends when the video distribution ends.

In step S200, the first image display information reception control unit 62 determines whether first image display information has been received from the mobile terminal 3. If the determination is affirmative, the process proceeds to step S202; if the determination is negative, the process enters a waiting state.

In step S202, the first image display information reception control unit 62 stores the received first image display information in the HDD 26.

In step S204, the second image display information generation unit 63 generates second image display information based on the received first image display information and stores it in the HDD 26.

In step S206, the second image display information transmission control unit 64 controls the external interface 32 to transmit the second image display information to a mobile terminal 3 different from the mobile terminal 3 that transmitted the first image display information, and then the process returns to step S200.

Note that if second image display information is already associated with the video to be distributed to the mobile device 3, the server 4 may also transmit the second image display information to the mobile device 3. This allows the user to recognize first images 51 displayed by other users in the past, rather than in real time. For example, even if a user registered in a group watches a video as part of group 5 and then watches the video alone (asynchronously with the other mobile devices 3) rather than as part of group 5, a second image 52 based on the previously displayed first image 51 will be displayed in the second image display area 50C.

As described above, the mobile device 3 according to the embodiment has a screen 3a configured with a distributed video display area 50A that displays video distributed from the server 4, a first image display area 50B that displays a first image 51 in response to a user's operation, and a second image display area 50C that displays a second image 52 based on a first image 51 displayed on another mobile device 3 by another user's operation. The first image 51 extends beyond the first image display area 50B and is superimposed on the distributed video display area 50A in response to the user's operation, but the second image 52 does not extend beyond the second image display area 50C and is superimposed on the distributed video display area 50A regardless of the other user's operation.

This allows the mobile device 3 to display the user's own reaction to the video on the screen 3a in a more entertaining manner, and also allows the user watching the video to see the reactions of other users without feeling annoyed.

Furthermore, the mobile terminal 3 according to the embodiment accepts a user's operation to display an onomatopoeia image indicating the user's emotion on the screen 3a while the screen 3a is displaying a video. The onomatopoeia image is then displayed on the screen 3a displaying the video, with the display mode changing depending on the user's operation mode. Therefore, the user can change the display mode of the onomatopoeia image displayed on the screen depending on the operation mode while watching content.

This allows the mobile device 3 to easily display the emotions felt while watching the video on the screen 3a while the user is watching the video.

3. Modifications
The technical scope of the present invention is not limited to the scope described in the above embodiments. Various modifications and improvements can be made to the above embodiments without departing from the spirit of the invention, and such modifications and improvements are also included in the technical scope of the present invention.

In the above embodiment, the user operates the display button 53 to display the first image 51 on the screen 3a, but the present invention is not limited to this. For example, the first image may be displayed on the screen 3a by the user operating the operation button 48 provided on the side of the mobile terminal 3, for example. Furthermore, if the mobile terminal 3 is equipped with a keyboard, the first image 51 may be displayed on the screen 3a by inputting a specified key.

Furthermore, in the above embodiment, the onomatopoeia display process is described as starting when the video is distributed and ending when the video is redistributed, but the present invention is not limited to this. For example, the onomatopoeia display process may be started a predetermined time before the video is distributed.

For example, if the start time of a video distribution is predetermined, the first image 51 and the second image 52 may be displayed in the first image display area 50B and the second image display area 50C by user operation from a predetermined time (e.g., 30 minutes) before the start time of the distribution. This allows the user to display their own reactions to the video, such as their expectations, on the screen 3a even before the start of the video distribution, and to check the reactions of other users. Furthermore, the first image 51 and the second image 52 may be displayed in the first image display area 50B and the second image display area 50C by user operation until a predetermined time (e.g., 30 minutes) after the end of the video distribution. This allows the user to display their own reactions to the video, such as their overall impressions, on the screen 3a even after the start of the video distribution, and to check the reactions of other users.

Furthermore, the positions and sizes of the streaming video display area 50A, first image display area 50B, second image display area 50C, and display button 53 on the screen 3a shown in Figure 4 etc. are also examples, and the present invention is not limited to these. For example, the configuration shown in Figure 11 may be used. The example of Figure 11(A) is an example of the display configuration of the screen 3a when the screen 3a is oriented horizontally (the long axis direction is horizontal to the user, and the short axis direction is vertical to the user). The example of Figure 11(B) is an example of the display configuration of the screen 3a when the screen 3a is oriented vertically (the long axis direction is vertical to the user, and the short axis direction is horizontal to the user).

In the example of FIG. 11(A), the streaming video display area 50A is in the center, with the first image display area 50B displayed to the right of it and the second image display area 50C displayed to the left. Then, multiple display buttons 53 are displayed vertically to the right of the first image display area 50B. In the example of FIG. 11(B), the streaming video display area 50A is displayed in the upper half of the screen 3a, the first image display area 50B is displayed on the right side of the lower half of the screen 3a, and the second image display area 50C is displayed on the left side. Note that in both FIGS. 11(A) and (B), the first image display area 50B may be displayed on the left side of the screen 3a and the second image display area 50C may be displayed on the right side, depending on the user's dominant hand.

In addition, in the above embodiment, the server 4 receives first image display information from the mobile terminal 3, transmits second image display information generated based on the first image display information to another mobile terminal 3, and the other mobile terminal 3 displays the second image 52 indicated by the second image display information in the second image display area 50C. However, the present invention is not limited to this.

For example, multiple mobile terminals 3 may send and receive first image display information to each other without going through the server 4, so that the mobile terminals 3 display the second image 52 in the second image display area 50C. In other words, when a mobile terminal 3 receives first image display information from another mobile terminal 3, it reads information about the first image 51 displayed on the other mobile terminal 3 from the first image display information, and displays a second image 52 based on that information in the second image display area 50C.

Furthermore, the various processing flows described in the above embodiments are merely examples, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the present invention.

1. Video distribution system 2. Communication line 3. Mobile terminal (an example of an "information processing device")
3a Screen (an example of a "display screen")
4. Server (an example of a "management means")
5 Groups (Examples of "Information Processing Systems")
50A Streaming video display area (an example of the “basic area”; an example of the “second part” of the first range)
50B First image display area (an example of the “first portion” of the first range)
50C Second image display area (an example of the “third part” of the first range, an example of the “second range”)
51 First image (an example of the "first element" or "additional element")
52 Second image (an example of the "second element")
53 Display button (an example of an "operation element")
70 Image display control unit (an example of the “first control means,” “second control means,” or “control means”)
71 Display area setting unit 72 First image display operation receiving unit (an example of “receiving means”)
73 Operation state determination unit 74 First image display information generation unit 75 First image display information transmission control unit 76 Second image display information reception control unit (an example of “acquisition means”)

Claims (4)

one or more processors;
The one or more processors:
Displaying the content in a content display area of a display unit included in the first information processing device;
displaying a first first element based on a first operation on the first information processing device in a first area at least partially overlapping with the content display area of the display unit, and displaying the first first element such that a position of the first first element moves based on a direction of the first operation;
displaying a second element based on a second operation on the second information processing device in a second area of the display unit;
ending the display of the second element on the first information processing device at a timing synchronized with a timing of ending the display of the display element displayed on the second information processing device based on the second operation;
Information processing device. one or more processors cause the content to be displayed in a content display area of a display unit included in the first information processing device;
one or more processors display a first first element based on a first operation by a first user who is viewing a viewing target on the first information processing device in a first area that at least partially overlaps the content display area of the display unit, and display the first first element such that a position of the first first element moves based on a direction of the first operation;
one or more processors cause a second element based on a second operation on the second information processing device to be displayed in a second area of the display unit;
one or more processors terminate the display of the second element on the first information processing device at a timing synchronized with a timing of termination of display of a display element displayed on the second information processing device based on the second operation;
Information processing methods. one or more processors,
a function of displaying the content in a content display area of a display unit included in the first information processing device;
a function of displaying a first first element based on a first operation on the first information processing device in a first area that at least partially overlaps with the content display area of the display unit, the function of displaying the first first element such that a position of the first first element moves based on a direction of the first operation;
a function of displaying a second element based on a second operation on the second information processing device in a second area of the display unit;
a function of ending the display of the second element on the first information processing device at a timing synchronized with a timing of ending the display of the display element displayed on the second information processing device based on the second operation;
A program to execute. A server and a terminal are provided,
The server
Displaying the content in a content display area of a display unit included in the first terminal;
displaying a first first element based on a first operation on the first terminal in a first area that at least partially overlaps with the content display area of the display unit, and displaying the first first element such that a position of the first first element moves based on a direction of the first operation;
displaying a second element based on a second operation on a second terminal in a second area of the display unit;
ending the display of the second element on the first terminal at a timing synchronized with a timing of ending the display of the display element displayed on the second terminal based on the second operation;
system.