JP7772080B2 - Information presentation system, device, method and program - Google Patents

Description

One aspect of this invention relates to an information presentation system that presents information presented at an online lecture or information exhibited at an online exhibition to viewers in the form of visual materials, as well as devices, methods, and programs used in this system.

In recent years, systems have been considered for online lectures and exhibitions, for example, that use visual materials such as text and diagrams for presentations and displays, such as posters and slides (see, for example, non-patent document 1). Such systems allow viewers to access the presentation or exhibition website from their own devices, select a desired poster by scrolling, and view its contents. Various mechanisms have also been proposed, such as enabling Q&A sessions between viewers and presenters via chat or web conference for each poster.

Rui Fukui and Tetsuharu Watanabe, "An attempt to make a poster-style academic conference online using Facebook as a platform," Transactions of the Japan Society of Mechanical Engineers, Vol. 87, No. 895, 2021

However, when viewing by scrolling, the information about the presentation or exhibition written on the poster is simply presented visually in a uniform manner, making it difficult for the presenter to fully convey the content of the presentation or exhibition to the viewer. Furthermore, while viewers can view the information in real time, they are simply reading the content of the presentation or exhibition, which does not always allow them to fully understand it. In other words, there are cases where information presentation is not satisfactory for both the presenter and the viewer.

On the other hand, using a system for question and answer sessions via chat or web conferencing makes it possible to present the content of a presentation or exhibition in more detail, but as the number of viewers increases, the burden on the presenter increases and the responsiveness of question and answer sessions decreases, making it impossible to maintain real-time communication.

This invention was made with the above-mentioned circumstances in mind, and aims to provide technology that enables information to be presented accurately in accordance with the viewer's viewing status while maintaining real-time performance when making a presentation or exhibition using online visual materials, thereby enabling information to be presented in a way that is highly satisfying for both the presenter and the viewer.

In order to solve the above-mentioned problems, one aspect of an information presentation system and its device or method according to the present invention transmits and presents explanatory information in a form of visual materials at a presentation or exhibition event online from an information presentation control device used by an information presenter to a viewer terminal used by a viewer via a network. The viewer terminal receives the explanatory information transmitted from the information presentation control device via the network and displays it on a display device, detects information regarding the viewer's viewing operation with respect to the displayed explanatory information, and transmits the detected information regarding the viewing operation to the information presentation control device via the network. In response, the information presentation control device acquires information regarding the viewing operation via the network, detects whether the scrolling direction is reverse based on the acquired information regarding the viewing operation, determines whether the scrolling in the reverse direction has been scrolled to the end, estimates based on the determination result whether the purpose of the scrolling in the reverse direction is "reconfirmation" or "supplement," generates auxiliary explanatory information including read-aloud information corresponding to the explanatory information depending on whether the purpose of the scrolling in the reverse direction is "reconfirmation" or "supplement," and transmits the generated auxiliary explanatory information to the viewer terminal via the network. The viewer terminal receives the transmitted auxiliary explanatory information via the network, and outputs the read-aloud audio information included in the received auxiliary explanatory information from an acoustic device.

According to one aspect of this invention, the viewer's viewing state is estimated based on the viewer's viewing operation on the presented explanatory information, and read-aloud audio information corresponding to the explanatory information is generated based on the estimated viewing state and output to the viewer. This allows the viewer to visually view the explanatory information on the poster while simultaneously audibly confirming the content through the read-aloud audio information.

Moreover, the content or audio source of the audio information to be read aloud is appropriately selected and output depending on the viewer's viewing status, such as the level of detail and purpose of the viewing. This makes it possible not only to simply read out explanatory information aloud, but also to present audio messages with content or audio sources that correspond to the viewer's viewing status, and furthermore, to present information in a more real-time manner than when using chat or web conferencing.

In other words, according to one aspect of this invention, when making a presentation or exhibition using online visual materials, it is possible to accurately present information according to the viewer's viewing state while maintaining real-time performance, thereby providing technology that enables the presentation of information in a manner that is highly satisfying for both the presenter and the viewer.

FIG. 1 is a diagram showing an example of the configuration of an information presentation system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the hardware configuration of a viewer terminal used by a viewer in the information presentation system shown in FIG. FIG. 3 is a block diagram showing an example of the software configuration of a viewer terminal used by a viewer in the information presentation system shown in FIG. FIG. 4 is a block diagram showing an example of the hardware configuration of a server device used as an information presentation control device by the information presenting side in the information presentation system shown in FIG. FIG. 5 is a block diagram showing an example of the software configuration of a server device used as an information presentation control device by the information presentation side in the information presentation system shown in FIG. FIG. 6A is a diagram showing an example of auxiliary explanatory information stored in the server device shown in FIG. 5 . FIG. 6B is a diagram showing an example of supplementary explanatory information stored in the server device shown in FIG. 5 . FIG. 7 is a diagram showing an example of visit history information stored in the server device shown in FIG. FIG. 8 is a flowchart showing an example of a processing procedure and processing content of a browsing control process executed by the browsing terminal shown in FIG. FIG. 9 is a flowchart showing an example of a processing procedure and processing content of the information presentation control processing executed by the server device shown in FIG. FIG. 10 is a flowchart showing an example of the procedure and content of the browsing state estimation process in the information presentation control process shown in FIG. FIG. 11 is a diagram showing an example of explanatory information in a format using visual materials distributed by a server device. FIG. 12 is a diagram showing an example of the explanatory information shown in FIG. 11 downloaded to a viewer terminal and displayed. FIG. 13 is a diagram showing a first example of a scroll operation on the explanation information displayed on the viewer terminal. FIG. 14 is a diagram showing a second example of a scroll operation on the explanation information displayed on the viewer terminal. FIG. 15 is a diagram showing a first example of a reference point for a scroll operation on explanation information displayed on a viewer terminal. FIG. 16 is a diagram showing a second example of a reference point for a scroll operation on explanation information displayed on a viewer terminal. FIG. 17 is a diagram showing a third example of a reference point for a scroll operation on explanation information displayed on a viewer terminal. FIG. 18 is a diagram showing an example of the positional relationship between a presenter, a viewer, their companions, and other participants in an exhibition. FIG. 19 is a diagram showing an example of setting the sound image position of the presenter in the positional relationship shown in FIG. FIG. 20 is a diagram showing an example of setting the sound image position of the accompanying person in the positional relationship shown in FIG. FIG. 21 is a diagram showing an example of setting the sound image positions of other participants in the positional relationship shown in FIG. FIG. 22 is a diagram for explaining a modified example of the output process of the auxiliary explanatory information.

The following describes an embodiment of the present invention with reference to the drawings.

[One embodiment]
In one embodiment of the present invention, an example will be described in which explanatory information about exhibits is presented in the form of visual materials such as letters and figures, such as posters and slides, at an online exhibition.

(Configuration example)
(1) System FIG. 1 is a diagram showing an example of the configuration of an information presentation system according to an embodiment of the present invention.

An information presentation system according to one embodiment includes a server device SV operated as an information presentation control device by the information presenting party, such as an exhibition organizer. This server device SV is accessible via a network NW by multiple exhibitor terminals ET1-ETm used by exhibitors of exhibits, and multiple viewer terminals UT1-UTn used by viewers of exhibits. Hereinafter, ET1-ETm and UT1-UTn will be abbreviated as ET and UT, respectively.

The exhibitor terminal ET consists of, for example, a personal computer, and has the function of uploading and registering explanatory information about exhibits created by exhibitors and their audio reading files to the server device SV via the network NW. The explanatory information explains the contents of the exhibit in a format using visual materials such as text and diagrams, such as posters and slides, and is created, for example, by combining text data with diagrams and photographs for each major category. The audio reading files represent the content of each major category included in the explanatory information by audio reading. The audio reading files include multiple audio files with different levels of granularity for the information content for each major category. The structure of the supplementary explanatory information will be explained in the operation example.

The network NW comprises, for example, a wide area network with the Internet at its core, and an access network for accessing this wide area network. Examples of access networks that can be used include public communication networks using wired or wireless connections, LANs (Local Area Networks) using wired or wireless connections, and CATV (Cable Television) networks.

(2) Devices (2-1) Viewer terminals UT1 to UTn
2 and 3 are block diagrams showing an example of the hardware configuration and software configuration of the viewer terminal UT, respectively.

The viewer terminal UT is, for example, a personal computer equipped with the input/output functions necessary for viewing information and a browser necessary for transmitting information data to and from the server device SV. Note that other devices such as tablet terminals and smartphones may also be used as the viewer terminal UT as long as they have similar functions.

The viewer terminal UT has a control unit 1 that uses a hardware processor such as a central processing unit (CPU). Connected to this control unit 1 are a storage unit having a program storage unit 2 and a data storage unit 3, a communication interface (hereinafter referred to as "I/F") 4, and an input/output I/F unit 5.

Under the control of the control unit 1, the communication I/F unit 4 performs data communication with the server device SV using a communication protocol defined by the network NW.

An input device 51, a display device 52, and a speaker 53 are connected to the input/output I/F unit 5. The input device 51 includes, for example, a keyboard and a mouse. The display device 52 is a device that uses liquid crystal or organic EL. The display device 52 is primarily used to visually display explanatory information about the exhibits downloaded from the server device SV. The input device 51 is primarily used to scroll the display position of the explanatory information displayed on the display device 52. The speaker 53 is used to amplified and output the read-aloud audio file sent from the server device SV.

In addition, a camera may be connected to the input/output I/F unit 5. By providing a camera, it is possible to detect the viewer's line of sight relative to the displayed explanatory information from, for example, the viewer's facial image, and recognize the viewer's viewing position relative to the explanatory information based on the detected line of sight.

The program storage unit 2 is configured, for example, by combining a non-volatile memory such as a solid-state drive (SSD) (storage medium) that can be written to and read from at any time with a non-volatile memory such as read-only memory (ROM), and stores middleware such as an operating system (OS), as well as application programs required to execute various control processes according to one embodiment. Hereinafter, the OS and each application program will be collectively referred to as the "program."

The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as RAM (Random Access Memory), and is equipped with an explanatory information storage unit 31 and an auxiliary explanatory information storage unit 32 as the main storage units required to implement one embodiment.

The explanatory information storage unit 31 is used to store explanatory information downloaded from the server device SV.

The auxiliary explanatory information storage unit 32 is used to store auxiliary explanatory information including audio files to be read aloud downloaded from the server device SV.

The control unit 1 includes, as processing functions necessary to implement one embodiment, an explanatory information reception processing unit 11, an explanatory information display processing unit 12, a scrolling operation detection processing unit 13, a browsing operation information transmission processing unit 14, an auxiliary explanatory information reception processing unit 15, and a reading voice output processing unit 16. These processing units 11 to 16 are all realized by causing the hardware processor of the control unit 1 to execute application programs stored in the program storage unit 2. Note that the application programs may be stored in advance in the program storage unit 2, or may be downloaded from the server device SV when needed and stored in the program storage unit 2.

The explanatory information receiving processing unit 11 accesses the server device SV in response to the viewer's viewing access operation, receives explanatory information about the exhibit from the server device SV via the communication I/F unit 4, and stores it in the explanatory information storage unit 31.

The explanatory information display processing unit 12 reads explanatory information from the explanatory information storage unit 31, and outputs and displays the read explanatory information to the display device 52 via the input/output I/F unit 5. At this time, it also performs processing to control the display position of the explanatory information in response to scrolling operations on the input device 51.

The scroll operation detection processing unit 13 performs processing to detect the amount and direction of the scroll operation on the displayed explanatory information at a predetermined detection period.

The browsing operation information transmission processing unit 14 generates browsing operation information including identification information representing the above-mentioned explanatory information and major item being viewed, and information representing the amount and direction of the detected scrolling operation, and performs processing to transmit the generated browsing operation information from the communication I/F unit 4 to the server device SV.

The auxiliary explanatory information receiving processing unit 15 receives auxiliary explanatory information corresponding to the explanatory information being viewed, which is sent from the server device SV, via the communication I/F unit 4, and performs processing to store it in the auxiliary explanatory information storage unit 32.

The reading audio output processing unit 16 reads the auxiliary explanatory information from the auxiliary explanatory information storage unit 32, converts the reading audio file contained in this auxiliary explanatory information into an analog audio signal, and then outputs it to the speaker 53 via the input/output I/F unit 5, thereby amplified and outputting the reading audio.

(2-2) Server device SV
4 and 5 are block diagrams showing an example of the hardware configuration and software configuration of the server device SV, respectively.

The server device SV consists of a server computer located, for example, on the cloud, and is equipped with a control unit 6 that uses a hardware processor such as a CPU. A storage unit having a program storage unit 7 and a data storage unit 8, and a communication I/F unit 9 are connected to this control unit 6 via a bus.

Under the control of the control unit 6, the communication I/F unit 9 sends and receives information data between the exhibitor terminal ET and the viewer terminal UT using a communication protocol defined by the network NW.

The program memory unit 7 is configured, for example, by combining a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read at any time with a non-volatile memory such as a ROM, and stores middleware such as an OS as well as programs necessary to execute various control processes in one embodiment of this invention.

The data storage unit 8 is, for example, a combination of a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read at any time, and a volatile memory such as RAM, and is equipped with an explanation information storage unit 81, a browsing operation information storage unit 82, a visit history information storage unit 83, a browsing status storage unit 84, and an auxiliary explanation information storage unit 85 as the main storage units required to implement one embodiment of this invention.

The explanatory information storage unit 81 is used to store explanatory information uploaded from the exhibitor terminal ET. As mentioned above, the explanatory information is created in a format that uses visual materials, for example, by combining text data with diagrams or photographs for each major category.

The browsing operation information storage unit 82 is used to store browsing operation information sent from the viewer terminal UT. As mentioned above, the browsing operation information includes the explanatory information being viewed, identification information indicating its major item, and information indicating the amount and direction of the scrolling operation.

The visit history information storage unit 83 is used to store the viewer's viewing history (e.g., number of views (also called visit history)) for the explanatory information being viewed by each major category based on the above-mentioned viewing operation information.

The viewing state memory unit 84 is used to store the estimated results of the viewer's viewing state for the explanatory information obtained by the control unit 6.

The auxiliary explanatory information storage unit 85 stores, for each major category of explanatory information, auxiliary explanatory information prepared in advance corresponding to each of multiple viewing states of the viewer that are expected while viewing the explanatory information. The auxiliary explanatory information storage unit 85 also stores, as auxiliary explanatory information, audio files representing the content of supplementary or advice that is actually provided or is expected to be provided by the viewer's companion or other participants while viewing the explanatory information.

The control unit 6 has, as processing functions according to one embodiment of the present invention, an explanatory information transmission processing unit 61, a viewing operation information acquisition processing unit 62, a viewing state estimation processing unit 63, an auxiliary explanatory information generation processing unit 64, and an auxiliary explanatory information transmission processing unit 65. These processing units 61 to 65 are all realized by causing the hardware processor of the control unit 6 to execute application programs stored in the program storage unit 7.

The explanatory information transmission processing unit 61 receives explanatory information uploaded from the exhibitor terminal ET via the communication I/F unit 9 and stores it in the explanatory information storage unit 81, and also reads out corresponding explanatory information from the explanatory information storage unit 81 in response to a viewing request from the viewer terminal UT, and transmits the read-out explanatory information from the communication I/F unit 9 to the requesting viewer terminal UT.

The viewing operation information acquisition processing unit 62 receives viewing operation information periodically transmitted from the viewer terminal UT while viewing the explanatory information via the communication I/F unit 9, and performs processing to store the received viewing operation information in the viewing operation information.

The browsing operation information acquisition processing unit 62 also performs processing to update the browsing history (visit history) by major category of explanatory information stored in the visit history information storage unit 83 based on the above-mentioned received browsing operation information.

Each time new browsing operation information is received, the browsing state estimation processing unit 63 reads the newly received browsing operation information and the most recently updated visit history information from the browsing operation information storage unit 82 and the visit history information storage unit 83, respectively. Then, the processing unit 63 performs processing to estimate the browsing state of the viewer based on the read browsing operation information and visit history information. This browsing state estimation processing will be described in the operation example.

Each time a new estimation result of the viewing state is obtained, the auxiliary explanatory information generation processing unit 64 searches the auxiliary explanatory information storage unit 85 using the estimation result and the viewing operation information used to obtain this estimation result as search conditions, and thereby performs a process of reading out the auxiliary explanatory information corresponding to the above search conditions.

The auxiliary explanatory information transmission processing unit 65 performs processing to transmit the read auxiliary explanatory information from the communication I/F unit 9 to the viewer terminal UT currently being viewed.

(Example of operation)
Next, an example of the operation of the information presentation system configured as above will be described.
Figure 8 is a flowchart showing the processing procedure and processing content of the viewing control processing executed by the control unit 1 of the viewer terminal UT, and Figure 9 is a flowchart showing the processing procedure and processing content of the information presentation control processing executed by the control unit 6 of the server device SV.

(1) Structure of the auxiliary explanatory information First, the structure of the auxiliary explanatory information will be described. Figures 6A and 6B are diagrams showing an example of the structure of the auxiliary explanatory information.
The auxiliary explanatory information corresponds, for each major item that constitutes explanatory information in a format using visual materials, to the identification information (major item ID) of that major item, multiple viewing states that are expected during viewing, sound sources corresponding to each viewing state, and a read-aloud audio file corresponding to the explanatory content of the major item.

In this example, the following five types of viewing states are defined: The first state is "skip" viewing, which corresponds to speed reading or skipping. The second state is "normal" viewing, which corresponds to viewing at a normal speed. The third state is "detailed" viewing, which corresponds to careful reading. The fourth state is "reconfirmation," in which after reaching the end of the page, the user returns to any major item on the page and views it again. The fifth state is "supplementary" explanation, in which the user returns to any major item in the middle of the page and views it again.

In this example, the sound sources are defined as "explainer," the viewer's "companion," and "other participants." The "explainer" is associated with the above-mentioned "skip," "normal," and "detailed" views. Furthermore, "reconfirmation" is associated with "other participants," and "companion" with "supplementary" explanation.

In this example, multiple audio files are prepared as audio readouts, with different levels of granularity in the explanation depending on the viewing status. For example, for "skip" viewing, an audio file is prepared that reads out only the "summary" of the explanation for each major item, for "normal" viewing, an audio file is prepared that reads out the explanation written in the major item as is, and for "detailed" viewing, an audio file is prepared that reads out the explanation for the major item and any additional explanations. Each of the above audio files is generated in advance by the exhibitor and uploaded from the exhibitor terminal ET.

In addition, for "reconfirmation," audio files are provided that provide supplementary explanations of major items or FAQs (Frequently Asked Questions) as read-alouds, and for "supplementary" explanations, audio files are provided for supplementary explanations by accompanying persons.

Furthermore, each reading audio file contains sound image position (also called sound image localization) designation information to represent the position of the sound source by the reading audio. The sound image position designation information consists of audio control data that controls the phase of the reading audio and sound image position display data that displays the position where the audio is being generated on the display screen of the viewer terminal UT.

(2) Presentation of explanatory information The explanation is given assuming that the explanatory information uploaded from the exhibitor terminal ET is stored in advance in the explanatory information storage unit 81 of the server device SV, and that the auxiliary explanatory information configured as described above is stored in advance in the auxiliary explanatory information storage unit 85.

Let's say that a viewer accesses the URL (Uniform Resource Locator) of the online exhibition on their viewer terminal UT in order to view exhibits at the online exhibition. Then, under the control of the explanatory information reception processing unit 11, the control unit 1 of the viewer terminal UT detects the access operation in step S10 shown in Figure 8, proceeds to step S11, and sends an access request to the server device SV of the exhibition site.

In response to this, when the server device SV receives the access request at step S30 shown in Figure 9 under the control of the explanatory information transmission processing unit 61, in step S31 it first reads the menu information of the exhibit from the explanatory information storage unit 81 and downloads it to the viewer terminal UT. As a result, the menu information of the exhibit is displayed on the display device 52 of the viewer terminal UT.

In this state, if a viewer specifies an exhibit they wish to view, the specified information for the exhibit is transmitted from the viewer terminal UT. In response, the control unit 6 of the server device SV, under the control of the explanatory information transmission processing unit 61, reads explanatory information for the poster corresponding to the specified information from the explanatory information storage unit 81, and downloads the read explanatory information to the viewer terminal UT that made the request. Figure 11 shows an example of the explanatory information to be downloaded. As shown in the figure, the explanatory information has multiple major items I1 to Ik, each of which is composed of, for example, a combination of text data and diagrams or photographs.

Under the control of the explanatory information reception processing unit 11, the control unit 1 of the viewer terminal UT receives the explanatory information in step S12 and temporarily stores the received explanatory information in the explanatory information storage unit 31. The control unit 1 then reads the received explanatory information from the explanatory information storage unit 31 and outputs the read explanatory information to the display device 52 via the input/output I/F unit 5. Thus, the explanatory information for the exhibit is displayed on the display device 52 of the viewer terminal UT from the top position of the page. Figure 12 shows an example of the display result of the explanatory information. Note that in this figure, ME indicates a margin area.

(3) Presentation of auxiliary explanatory information (3-1) Detection of viewing operation When the viewer performs a scrolling operation using the input device 51 while the above explanatory information is displayed, the explanatory information displayed on the display device 52 is scrolled, for example, as shown in SC of Figure 12.

When the control unit 1 of the viewer terminal UT detects the scrolling operation in step S13 under the control of the scrolling operation detection processing unit 13, it detects the amount and direction of the scrolling operation at a preset detection period (for example, any period less than one second) in step S14. Then, under the control of the browsing operation information transmission processing unit 14, it generates browsing operation information in step S15, including identification information indicating the explanatory information and main item being viewed, and the detected amount and direction of the scrolling operation, and transmits the generated browsing operation information from the communication I/F unit 4 to the server device SV.

In order for the server device SV to be able to identify the viewer's focus position on the explanatory information after the scroll operation, the scroll operation detection processing unit 13 sets a reference point for the focus position. For example, the following method is used to set this reference point.

The first technique assumes that the viewer is focusing on position P0, a predetermined distance Lth below the top edge of the currently displayed page, as shown in Figure 15, and uses this position as the reference point for scrolling.

The second technique, as shown in Figure 16, assumes that the position pointed to by the mouse pointer or cursor CU is the position P0 at which the viewer is focusing, and uses the coordinates of that position as the reference point for scrolling.

The third method, when a camera is provided, detects the viewer's line of sight L from the viewer's facial image captured by the camera, as shown in Figure 17, and calculates the viewer's focus position P0 based on this line of sight L. The position coordinates of the calculated focus position P0 are then used as the reference point for scrolling.

The coordinates P0 of the scrolling reference point set as described above are inserted into the viewing operation information along with the detection results of the amount and direction of scrolling and are sent to the server device SV.

When the browsing operation information is transmitted from the viewer terminal UT, the control unit 6 of the server device SV, under the control of the browsing operation information acquisition processing unit 62, receives the browsing operation information transmitted from the viewer terminal UT via the communication I/F unit 9 in step S32 and temporarily stores the received browsing operation information in the browsing operation information storage unit 82. At the same time, in step S33, the browsing operation information acquisition processing unit 62 updates the number of views (number of visits) for the corresponding major item stored in the visit history information storage unit 83 based on the identification information of the major item being viewed, which is included in the browsing operation information. Figure 7 shows an example of the updated browsing history information.

(3-2) Estimation of Viewing State When the above viewing operation information is received, the control unit 6 of the server device SV, under the control of the viewing state estimation processing unit 63, executes a process of estimating the viewing state of the viewer with respect to the explanation information in step S34 as follows.

FIG. 10 is a flowchart showing the processing procedure and processing contents of the browsing state estimation processing.
That is, in step S41, the viewing state estimation processing unit 63 reads the amount and direction of the scrolling operation contained in the viewing operation information and the reference position coordinates of the scrolling from the viewing operation information storage unit 82, and further reads the number of views (number of visits) for the major item currently being viewed from the visit history information storage unit 83.

Once the above loading is complete, the viewing state estimation processing unit 63 first determines in step S43 whether the direction of the scrolling operation SC is "forward" or "reverse." If the result of this determination is that the scrolling direction is "forward," as shown in FIG. 13, for example, the viewing state estimation processing unit 63 calculates the speed Vcur of the scrolling operation based on the amount of scrolling operation. The speed Vcur of the scrolling operation is calculated, for example, as an average value over a predetermined period (for example, 1 second).

The viewing state estimation processing unit 63 then compares the calculated scrolling speed Vcur with a first threshold value Vth1 in step S44. If the result of this comparison is Vcur ≧ Vth1, the scrolling speed is fast and can be considered to be speed reading or skipping, and therefore in step S46 the viewing state at this time is estimated to be "skip" viewing.

On the other hand, if Vcur ≧ Vth1 is not true, the viewing state estimation processing unit 63 compares the scrolling speed Vcur with a second threshold value Vth2 (Vth2 < Vth1) in step S45. If the comparison shows that Vcur > Vth2, this can be considered to correspond to browsing at a normal speed, and therefore the viewing state at this time is estimated to be "normal" viewing in step S47. On the other hand, if Vcur > Vth2 is not true, the scrolling speed is slow and can be considered to correspond to a state of thorough reading, and therefore the viewing state estimation processing unit 63 estimates the viewing state at this time to be "detailed" viewing in step S48.

On the other hand, suppose that in step S43 above, the scrolling direction is determined to be "reverse," as shown in FIG. 14, for example. In this case, in step S50, the viewing state estimation processing unit 63 determines whether the page has been scrolled to the end based on the viewing history information. If this determination shows that the page has been scrolled to the end, the viewing state estimation processing unit 63 infers in step S51 that the current viewing state is "reconfirmation," in which the user reaches the end of the page and then returns to any major item within the page to view it again. On the other hand, if the page has not been scrolled to the end, the processing unit 63 infers in step S52 that the current viewing state is "supplementary" explanation, in which the user returns to any major item within the page and views it again.

(3-3) Generation of auxiliary explanatory information When the above-mentioned viewing state estimation process is completed, the control unit 6 of the server device SV then, in step S35, executes the auxiliary explanatory information generation process as follows under the control of the auxiliary explanatory information generation processing unit 64.

That is, the auxiliary explanatory information generation processing unit 64 acquires the estimation result of the browsing information from the browsing state estimation processing unit 63, and reads the browsing operation information and visit history information used to obtain the estimation result from the browsing operation information storage unit 82 and the visit history information storage unit 83, respectively. Then, the auxiliary explanatory information storage unit 85 is searched using the estimation result of the browsing state and the browsing operation information as search conditions, thereby reading out auxiliary explanatory information corresponding to the search conditions.

For example, suppose the current position of interest (viewing position) has major item ID = 1 and the estimated viewing state is "skip" viewing. In this case, the auxiliary explanatory information generation processing unit 64 reads the sound source "explainer" and the audio file "1-1.wav" corresponding to the above search criteria from the auxiliary explanatory information (e.g., Figure 6A) stored in the auxiliary explanatory information storage unit 85. At this time, the audio file "1-1.wav" contains a low-granularity read-aloud audio file that reads out only the "summary" of the explanatory content, corresponding to the above "skip" viewing. The audio file "1-1.wav" also contains sound image position designation information for representing the position of the sound source, the "explainer." This sound image position designation information is composed of audio control data that controls the phase of the read-aloud audio and sound image position display data that displays the audio generation position on the display screen.

Also, for example, suppose the viewing position is major item ID = 1 and the estimated viewing state is "normal" viewing. In this case, the auxiliary explanatory information generation processing unit 64 reads out the sound source "explainer" and audio file "1-2.wav" corresponding to the above search conditions from the auxiliary explanatory information (e.g., Figure 6A) stored in the auxiliary explanatory information storage unit 85. At this time, the audio file "1-2.wav" contains a medium-granularity read-out audio file that reads out the explanatory content written in the major item as is, corresponding to the above "normal" viewing. Also, in this case, since the sound source is "explainer," the audio file "1-2.wav" contains sound image position designation information for expressing the position of the "explainer."

Furthermore, suppose the viewing position is major item ID = 1 and the estimated viewing state is "detailed" viewing. In this case, the auxiliary explanatory information generation processing unit 64 reads out the sound source "explainer" and audio file "1-3.wav" corresponding to the above search conditions from the auxiliary explanatory information (e.g., Figure 6A) stored in the auxiliary explanatory information storage unit 85. At this time, the audio file "1-3.wav" contains a highly granular read-aloud audio file that reads out the explanation content of the major item and the content of the additional explanation corresponding to the above "detailed" viewing. Also, in this case, since the sound source is "explainer," the audio file "1-3.wav" contains sound image position designation information for expressing the position of the "explainer."

On the other hand, suppose the viewing position has major item ID = 1 and the estimated viewing state is "supplemental" explanation. In this case, the auxiliary explanation information generation processing unit 64 reads out the sound source "companion" and the audio file "1-sup1.wav" or "1-sup2.wav" that correspond to the above search criteria from the auxiliary explanation information (e.g., Figure 6B) stored in the auxiliary explanation information storage unit 85. At this time, the audio file "1-sup1.wav" or "1-sup2.wav" contains an audio file reading out the supplemental explanation for the companion. Furthermore, since the sound source is "companion," the audio file "1-sup1.wav" or "1-sup2.wav" contains sound image position designation information for expressing the position of the "companion."

The reason two audio files, "1-sup1.wav" and "1-sup2.wav," are provided is to accommodate cases where there are two accompanying persons. These audio files are registered in advance and linked to the identification information of the accompanying person who has been registered in advance. The accompanying person's audio file may be obtained from the accompanying person's terminal (not shown) and registered as needed during viewing.

Also, for example, suppose the viewing position is major item ID = 1 and the estimated viewing state is "Reconfirm." In this case, the auxiliary explanatory information generation processing unit 64 reads the sound source "Other Participants" and audio file "1-con1.wav" corresponding to the above search criteria from the auxiliary explanatory information (e.g., Figure 6B) stored in the auxiliary explanatory information storage unit 85. At this time, the audio file "1-con1.wav" contains an audio file that provides supplementary explanations or FAQs to the explanation content of the major item, corresponding to the above "Reconfirm." Furthermore, since the sound source is "Other Participants," the audio file "1-sup1.wav" contains sound image position designation information for representing the position of "Other Participants." Note that in this case, assuming that there are multiple other participants, audio files corresponding to each of the multiple participants are prepared.

(3-4) Output of supplementary explanatory information In step S36, under the control of the supplementary explanatory information transmission processing unit 65, the control unit 6 of the server device SV transmits the supplementary explanatory information generated by the supplementary explanatory information generation processing unit 64 from the communication I/F unit 9 to the viewer terminal UT.

In response to this, when the control unit 1 of the viewer terminal UT detects the receipt of auxiliary explanatory information in step S16, under the control of the auxiliary explanatory information reception processing unit 15, in step S17, the control unit 1 temporarily stores the auxiliary explanatory information received by the communication I/F unit 4 in the auxiliary explanatory information storage unit 32. Next, under the control of the reading audio output processing unit 16, the control unit 1 of the viewer terminal UT reads a reading audio file from the stored auxiliary explanatory information. Then, the read reading audio file is converted into an analog audio signal, which is then output to the speaker 53 via the input/output I/F unit 5 and amplified as audio.

As a result, for example, if the audio file included in the auxiliary explanatory information corresponds to "skip" viewing, a read-out audio is output that indicates an "overview" of the explanatory content of the major item being viewed. Also, for example, if the audio file included in the auxiliary explanatory information corresponds to "normal" viewing, a read-out audio is output that indicates the explanatory content written in the major item being viewed. Furthermore, for example, if the audio file included in the auxiliary explanatory information corresponds to "detailed" viewing, a read-out audio is output that indicates the explanatory content of the major item being viewed and the content of any additional explanations.

In addition, at this time, the auxiliary explanatory information receiving processing unit 15 expresses the position of the sound source as follows by controlling the phase of the reading voice in accordance with the sound source position designation information contained in the audio file.

In other words, let's assume that the positional relationship of the presenter ES, viewer US, their companion AS, and other participants PS with respect to the exhibits at an actual exhibition is, for example, as shown in Figure 18. In this case, if the sound source is the "presenter," the phase of the reading audio is controlled so that it sounds as if the audio is being read aloud in front of the exhibits. Also, if the sound source is the "companion," the phase of the reading audio is controlled so that it sounds as if the audio is being read aloud to the side of the viewer. Furthermore, if the sound source is the "other participants," the phase of the reading audio is controlled so that it sounds as if the audio is being read aloud behind the viewer.

Furthermore, the auxiliary explanatory information receiving processing unit 15 determines in step S19 whether the sound source position designation information included in the auxiliary explanatory information includes sound source position display data. If sound source position display data is included, then, under the control of the explanatory information display processing unit 12, in step S20, the sound source position display data is superimposed on the explanatory information and displayed on the display device 52.

For example, if the sound source is the "presenter," sound source position SL1 is visually displayed on the top edge of the margin area ME of the display screen, as shown in Figure 19. If the sound source is the "accompanying person," sound source position SL2 is visually displayed on the side edge of the margin area ME of the display screen, as shown in Figure 20. If the sound source is the "other participant," sound source position SL3 is visually displayed on the bottom edge of the margin area ME of the display screen, as shown in Figure 21. All of the sound source positions SL1, SL2, and SL3 are displayed in the form of, for example, an animation of spreading sound waves.

By controlling the phase of the read-out audio as described above, the position of the sound source is audibly represented, and the position of the sound source is also visually displayed on the display screen, allowing viewers to get a sense of the position and distance of the presenter, companions, and other participants through the read-out audio and display, even while viewing online. In other words, it is possible to achieve a realistic online viewing experience.

In addition, when the viewer terminal UT and server device SV detect the end of viewing in steps S21 and S37, respectively, they terminate the series of information presentation processes.

(Actions and Effects)
As described above, in one embodiment, while a viewer is online viewing explanatory information in the form of visual materials such as characters and figures, such as posters and slides, the viewer terminal UT detects the viewer's scrolling operation and transmits the amount and direction of the scrolling to the server device SV. Meanwhile, the server device SV estimates the viewer's viewing state based on the amount and direction of the scrolling operation, generates read-aloud audio files of different granularity corresponding to the content of the explanatory information according to the estimated viewing state, transmits these to the viewer terminal UT as auxiliary explanatory information, and outputs the read-aloud audio to the viewer in a loud voice.

Therefore, in addition to the visually displayed explanatory information, viewers can also confirm the content of the explanatory information through audio reading. Moreover, audio reading with different information content and granularity is output depending on the viewer's viewing state, allowing viewers to confirm the desired content and granularity of information through audio reading depending on their own viewing state. Therefore, viewers can confirm information with the desired granularity in real time without using chat or web conferencing.

In addition, sound source location specification information is added to the reading audio file, which controls the phase of the reading audio and visually displays the sound source location. As a result, even while viewing online, viewers can sense the location and distance of the sound source, such as the presenter, accompanying persons, and other participants, allowing for a more realistic viewing experience.

[Other embodiments]
(1) In the above embodiment, the system detects which major item the viewer focuses on through a scroll operation, and outputs a read-out audio corresponding to the explanatory content of the major item where the focus is located. However, this is not limited to this. For example, as shown in FIG. 22 , a margin area ME may be set between major items Ik-1 and Ik, and when the viewer's viewing position through a scroll operation SC moves from P5 on major item Ik-1 to position P6 on the margin area ME, a read-out audio corresponding to the content of the explanatory information of major item Ik-1 before the movement may be output. In this case, the width of the margin area ME may be fixed in advance, or may be set to, for example, a value obtained by multiplying the average length of the writing areas of major items Ik-1 and Ik by a specific multiplier, or a value obtained by multiplying the average amount of scrolling by the user per scroll by a specific multiplier.

(2) When switching the output of the reading audio in response to the movement of the focus position due to scrolling, the output of the next reading audio may be controlled so that the current reading audio is output until it has reached its end. In this way, the reading audio is not interrupted midway. When switching the output of the reading audio, effect processing such as fade-in/fade-out may be performed.

(3) In the above embodiment, the viewing state was estimated based on the scrolling speed, but the viewing state may also be estimated based on factors other than the scrolling speed, such as the length of time the mouse pointer or cursor is stopped.

(4) In the above embodiment, the auxiliary explanatory information is registered in advance in the auxiliary explanatory information storage unit 85, and the corresponding auxiliary explanatory information is read out depending on the estimated result of the viewing state, etc. However, this is not limited to this, and a learning model constructed using deep learning, etc. may be prepared, and the feature quantities of the viewing operation may be input to this learning model as explanatory variables, and the audio file of the auxiliary explanatory information or its label may be output as the objective variable.

(5) In the above embodiment, the case where the viewing state estimation process is performed on the server device SV has been described as an example. However, this is not limited to this, and the viewing state estimation process may be performed on the viewer terminal UT and the estimation result may be transmitted to the server device SV.

(6) In the above embodiment, the explanatory information and auxiliary explanatory information are stored in advance in the server device SV, and the explanatory information and auxiliary explanatory information are transmitted from the server device SV to the viewer terminal UT in response to the viewer's viewing operations. However, this is not limited to this. For example, the explanatory information and additional explanatory information may be stored in the exhibitor terminal ET, and the explanatory information and auxiliary explanatory information may be transmitted to the viewer terminal UT directly from the exhibitor terminal ET or via the server device SV in response to the viewer's viewing operations.

(7) While the above embodiment was described using an online exhibition as an example, the present invention may also be applied to systems for conducting online lectures and presentations, and may also be applied to websites that present information according to a certain purpose and context, such as public relations pages for companies and educational institutions, or product introduction pages in online stores. For example, in the case of a product introduction page, the presenter would be a company's product planner or developer, and the other participants would be users who have actually used the product.

(8) In addition, various modifications can be made to the method of detecting viewing operations, the method of estimating viewing states, the types of viewing states, the configuration of the audio file to be read, the configuration of the explanatory information, and the configuration and functions of the viewer terminal and the information presentation control device without departing from the spirit of this invention.

Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations corresponding to the embodiments may be adopted as appropriate.

In short, this invention is not limited to the above-described embodiments, and can be embodied by modifying the components in the implementation stage without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be deleted from all of the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

SV...server device UT1 to UTn...viewer terminal ET1 to ETm...exhibitor terminal NW...network 1...control unit of viewer terminal 2...program storage unit of viewer terminal 3...data storage unit of viewer terminal 4...communication I/F unit of viewer terminal 5...input/output I/F unit of viewer terminal 6...control unit of server device 7...program storage unit of server device 8...data storage unit of server device 9...communication I/F unit of server device 11...explanatory information reception processing unit 12...explanatory information display processing unit 13...scrolling operation detection processing unit 14...viewing operation information transmission processing unit 15...auxiliary explanatory information reception processing unit 16...reading voice output processing unit 31...explanatory information storage unit 32...auxiliary explanatory information storage unit 61...explanatory information transmission processing unit 62...viewing operation information acquisition processing unit 63...viewing state estimation processing unit 64...auxiliary explanatory information generation processing unit 65...auxiliary explanatory information transmission processing unit 81...explanatory information storage unit 82: Viewing operation information storage unit 83: Visit history information storage unit 84: Viewing state storage unit 85: Supplementary explanation information storage unit

Claims (7)

An information presentation system that transmits explanatory information in a form using visual materials at a presentation or exhibition event from an information presentation control device used by an information presenter to a viewer terminal used by a viewer online via a network and presents the information,
The viewer terminal includes:
a display processing unit that receives the explanatory information transmitted from the information presentation control device via the network and displays the received explanatory information on a display device;
a viewing operation transmission processing unit that detects information regarding a viewing operation of the viewer with respect to the explanation information displayed on the display device, and transmits the detected information regarding the viewing operation to the information presentation control device via the network,
The information presentation control device includes:
an estimation processing unit that acquires information about the browsing operation via the network, detects whether the scrolling direction is reverse based on the acquired information about the browsing operation, and determines whether the scrolling in the reverse direction has been completed to the end, and estimates whether the purpose of the scrolling in the reverse direction is "reconfirmation" or "supplement" based on the determination result ;
a generation processing unit that generates supplementary explanatory information including voice reading information corresponding to the explanatory information depending on whether the purpose of the backward scrolling is "reconfirmation" or "supplementation" ;
an auxiliary explanatory information transmission processing unit that transmits the generated auxiliary explanatory information to the viewer terminal via the network,
The viewer terminal includes:
An information presentation system comprising an audio output processing unit that receives the auxiliary explanatory information via the network and outputs the read-aloud audio information included in the received auxiliary explanatory information from an acoustic device. An information presentation control device that transmits explanatory information in a form using visual materials at a presentation or exhibition event to a viewer terminal used by a viewer via a network and presents the information,
an estimation processing unit that acquires information about the viewer's viewing operation for the presented explanation information from the viewer terminal via the network, detects whether the scrolling direction is reverse based on the acquired information about the viewing operation, and determines whether the scrolling in the reverse direction has been completed to the end, and estimates whether the purpose of the scrolling in the reverse direction is "reconfirmation" or "supplement" based on the determination result ;
a generation processing unit that generates supplementary explanatory information including voice reading information corresponding to the explanatory information depending on whether the purpose of the backward scrolling is "reconfirmation" or "supplementation" ;
a transmission processing unit that transmits the generated auxiliary explanatory information to the viewer terminal via the network and causes the viewer terminal to output the read-aloud audio information. the estimation processing unit estimates a browsing detail level of the viewer for the explanation information based on information related to the browsing operation ;
the generation processing unit generates the read-aloud voice information with different information granularity corresponding to the explanation information based on the estimated browsing detail level;
The information presentation control device according to claim 2 , wherein the transmission processing unit transmits the auxiliary explanatory information including the generated read-aloud voice information to the viewer terminal, and causes the read-aloud voice information to be output from the viewer terminal. the estimation processing unit estimates a viewing purpose of the viewer for the explanation information based on information related to the viewing operation ;
the generation processing unit selects a corresponding sound source from a plurality of explanation sound sources prepared in advance based on the estimated viewing purpose, and generates the reading voice information reflecting a sound image position of the selected sound source;
The information presentation control device according to claim 2 , wherein the transmission processing unit transmits the auxiliary explanatory information including the generated read-aloud voice information to the viewer terminal, and causes the read-aloud voice information to be output from the viewer terminal. the generation processing unit further generates sound image position display information representing the sound image position;
The transmission processing unit transmits the generated auxiliary explanatory information including the sound image position display information to the viewer terminal, and displays the sound image position display information on the viewer terminal in a state where the sound image position display information is superimposed on the explanatory information.
The information presentation control device according to claim 4 . 1. An information presentation method implemented by an information presentation system that transmits explanatory information in a form using visual materials at a presentation or exhibition event from an information presentation control device used by an information presenter to a viewer terminal used by a viewer online via a network, comprising:
receiving, by the viewer terminal, the explanatory information transmitted from the information presentation control device via the network, and displaying the received explanatory information on a display device;
a step of detecting, by the viewer terminal, information regarding a viewing operation of the viewer with respect to the explanatory information displayed on the display device, and transmitting the detected information regarding the viewing operation to the information presentation control device via the network;
a step of acquiring information about the browsing operation via the network by the information presentation control device, detecting whether the scrolling direction is reverse based on the acquired information about the browsing operation, and determining whether the scrolling in the reverse direction has been completed to the end, and estimating whether the purpose of the scrolling in the reverse direction is "reconfirmation" or "supplement" based on the determination result ;
generating, by the information presentation control device, auxiliary explanatory information including voice reading information corresponding to the explanatory information, depending on whether the purpose of the backward scrolling is "reconfirmation" or "supplement";
a step of transmitting the generated auxiliary explanatory information to the viewer terminal via the network by the information presentation control device;
receiving the auxiliary explanatory information via the network by the viewer terminal, and outputting the read-aloud audio information contained in the received auxiliary explanatory information from an acoustic device. 6. A program causing a processor included in the information presentation control device to execute processing by each of the processing units included in the information presentation control device according to claim 2 .