JP7629036B2 - Appreciation support system, appreciation support method, and appreciation support program - Google Patents

Description

Article 30, Paragraph 2 of the Patent Act applied. Exhibited at the CEATEC 2022 general exhibition held at Makuhari Messe from October 18 to 21, 2022.

This disclosure relates to an appreciation support system, an appreciation support method, and an appreciation support program for assisting in the appreciation of exhibits.

Some art galleries and museums have introduced audio guides that provide visitors with audio commentary about the exhibits. In this case, visitors carry a rented earphone device. When the visitor enters the exhibit number into the earphone device, the audio guide is played.

Guidance devices using characters are also under consideration (for example, Patent Document 1). The guidance device described in this patent document has a memory unit that stores explanation data in which each of multiple characters explains the exhibits. The device then accepts a selection from the user of the character that will explain the exhibits. It then outputs the explanation data corresponding to the selected character. Furthermore, the guidance device generates a composite image by combining an image of the user captured by a camera device, a character image corresponding to the selected character, and an image of the user captured by the camera device.

JP 2020-71519 A

In the technology described in the patent document, a character reacts based on the user's answers. However, the user's answers alone are not enough to increase the user's interest.

The viewing support system that solves the above problem includes a control unit connected to an information acquisition device. The control unit acquires the reaction of the viewer of the exhibit from the information acquisition device, and determines and outputs the agent's response in response to the reaction based on the characteristic information extracted from the exhibit.

This disclosure makes it possible to provide information that will increase a user's interest in the exhibits.

FIG. 1 is an explanatory diagram of an appreciation support system according to an embodiment. FIG. 2 is an explanatory diagram of a hardware configuration of the embodiment. FIG. 2 is an explanatory diagram of a processing procedure according to an embodiment. FIG. 2 is an explanatory diagram of a processing procedure according to an embodiment.

Below, an embodiment of an art appreciation support system, an art appreciation support method, and an art appreciation support program will be described with reference to Figures 1 to 4. In this embodiment, a service is assumed that provides exhibit information tailored to the user about paintings exhibited at an art museum.

As shown in FIG. 1, the viewing support system of this embodiment uses a camera 10, a microphone 11, a monitor 15, and a support server 20, which are connected via a network. The camera 10 and the microphone 11 function as information acquisition devices that acquire the situation of the viewer U1.

The camera 10 captures an image of a viewer U1 viewing an exhibit E1.
The microphone 11 collects the speech of the viewer U1.
The monitor 15 is placed in the vicinity of the exhibit E1 in correspondence with it. Information relating to the exhibit E1 is output to this monitor 15. An agent avatar image 151 and an analysis result screen 152 are output to this monitor 15. Furthermore, the monitor 15 is equipped with a speaker, and outputs the agent's message by voice synthesis.

(Hardware configuration example)
FIG. 2 shows an example of the hardware configuration of an information processing device H10 that functions as the support server 20 or the like.

The information processing device H10 has a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H15. Note that this hardware configuration is an example, and other hardware may be included.

The communication device H11 is an interface that establishes a communication path with other devices and transmits and receives data, such as a network interface or a wireless interface.

The input device H12 is a device that accepts input from a user, etc., and is, for example, a mouse or a keyboard. Furthermore, in this embodiment, the input device H12 also includes the camera 10 and microphone 11 described above. The display device H13 is a display or touch panel that displays various information, and in this embodiment, also includes the monitor 15 described above.

The storage device H14 is a storage device that stores data and various programs for executing various functions of the support server 20. Examples of the storage device H14 include ROM, RAM, a hard disk, etc.

The processor H15 uses the programs and data stored in the storage device H14 to control each process in the support server 20 (for example, the process in the control unit 21 described below). Examples of the processor H15 include a CPU and an MPU. The processor H15 expands a program stored in a ROM or the like into a RAM and executes various processes corresponding to the various processes. For example, when an application program of the support server 20 is started, the processor H15 operates a process that executes each process described below.

Processor H15 is not limited to performing software processing for all of the processes it executes. For example, processor H15 may be equipped with a dedicated hardware circuit (e.g., an application specific integrated circuit: ASIC) that performs hardware processing for at least some of the processes it executes. That is, processor H15 may be configured as follows:

A circuit includes: (1) one or more processors that operate according to a computer program; (2) one or more dedicated hardware circuits that perform at least some of the processes; or (3) a combination thereof. A processor includes a CPU and memory, such as RAM and ROM, that stores program code or instructions configured to cause the CPU to perform processes. Memory, i.e., computer-readable media, includes any available media that can be accessed by a general-purpose or special-purpose computer.

(Functions of Support Server 20)
The function of the support server 20 will be described with reference to FIG.
The support server 20 is a computer system for providing information about exhibits to the viewer U1. The support server 20 includes a control unit 21, an exhibition information storage unit 22, and an analysis information storage unit 23.

The control unit 21 performs the processes described below (processing including an acquisition stage, an analysis stage, an output control stage, etc.). By executing a viewing support program for this purpose, the control unit 21 functions as an acquisition unit 211, an analysis unit 212, an output control unit 213, etc.

3, the acquisition unit 211 acquires a captured image of the viewer U1 from the camera 10. From this captured image, the facial expression and actions (pointing direction, etc.) of the viewer U1 can be identified.
Furthermore, the acquisition unit 211 acquires the voice uttered by the viewer U1 from the microphone 11.

The analysis unit 212 in FIG. 1 calculates the number of viewers based on the image of the viewer U1. The analysis unit 212 also analyzes the facial expressions of each viewer U1. Furthermore, the analysis unit 212 performs co-occurrence analysis using keywords acquired from the viewer U1.

As shown in FIG. 3, the analysis unit 212 performs an analysis according to the exhibit, depending on the relationship between the characteristic information of the exhibit E1 and the reaction of the viewer U1. The reaction of the viewer U1 includes the viewer's speech and gestures, etc. For example, the analysis unit 212 identifies the designated area of the exhibit E1 that the viewer U1 is aware of, depending on the position where the viewer U1 is pointing. Furthermore, the analysis unit 212 analyzes the viewer U1's emotions toward the exhibit E1 by performing voice recognition of the viewer U1's speech. Emotions may also be analyzed by emotion recognition using voice and facial expressions.

1 controls the output of a response according to the reaction of the viewer U1. This response is output using the movement of the avatar image 151 and the voice of the agent.
As shown in FIG. 3, the output control unit 213 outputs the analysis results by the agent on the monitor 15. In this case, responses using avatar images or voice synthesis are used. Furthermore, the output control unit 213 adjusts the output timing of the agent's response according to the movements and facial expressions of the viewer U1. For example, when the output control unit 213 detects a debating facial expression of the viewer U1, the agent response is put on hold until the debating is completed. Furthermore, when multiple viewers U1 are detected from the captured image, the viewer U1 to be spoken to is specified and the agent response is output. In this case, the viewer U1 to be spoken to is specified among the multiple viewers U1 by using the viewer U1's standing position, the pattern or color of his/her clothing, etc.

The exhibition information storage unit 22 records exhibition management information related to exhibits exhibited at the venue. This exhibition management information is recorded when an exhibit E1 is installed at the venue. This exhibition management information includes data related to the exhibit ID, attributes, and characteristic information.

The exhibit ID is an identifier for identifying each exhibit E1 (painting).
The attributes are information about the title of the exhibit, the artist, etc. The information about the artist includes the artist's name and contact information.

The feature information includes the features of exhibit E1 relative to the display position of this exhibit. For example, if exhibit E1 is a flat object such as a painting, information (keywords) relating to the pattern and color of the flat object's two-dimensional shape (spatial coordinates) is recorded as feature information. If exhibit E1 is a three-dimensional object such as a sculpture, information (keywords) relating to the pattern and color of the three-dimensional object's three-dimensional shape (spatial coordinates) is recorded as feature information. By using this feature information, the indicated area of exhibit E1 can be identified depending on the direction in which viewer U1 is pointing.

The analysis information storage unit 23 records analysis management information related to the reactions of viewers. This analysis management information is recorded when the reaction of viewer U1 is analyzed. This user management information includes data related to the exhibit ID, user ID, and analysis results.

The exhibit ID is an identifier for identifying each exhibit E1.
The user ID is an identifier for identifying the viewer U1 of the exhibit. For example, information that can identify the viewer (for example, features of a facial image) or the date and time of visit can be used as the user ID.

The analysis results are the results of an analysis based on the user's reactions. In this embodiment, the analysis results are recorded in association with keywords extracted from the facial expressions and gestures of the viewer U1, and words (keywords) included in the speech of the viewer U1. These keywords are then used to perform a co-occurrence analysis to analyze combinations of words that often appear together in a sentence.

(Viewing Support Processing)
Next, the viewing support process will be described with reference to FIG.
First, the control unit 21 of the support server 20 executes a process for determining the viewer's state (step S11). Specifically, the acquisition unit 211 of the control unit 21 acquires an image captured by the camera 10. This image includes, as a subject, one or more viewers U1 viewing the exhibit E1. The analysis unit 212 then calculates the number of viewers U1 included in the captured image. Furthermore, the analysis unit 212 analyzes the state of consciousness of each viewer U1 by image recognition of each viewer U1. As for the state of consciousness, the analysis unit 212 analyzes whether the viewer U1 is considering the exhibit E1, for example, by using his/her imagination or by being immersed in the exhibit.

Next, the control unit 21 of the support server 20 executes an agent operation process according to the viewer's situation (step S12). Specifically, the output control unit 213 of the control unit 21 outputs an avatar image 151 to the monitor 15. This avatar image 151 is displayed facing the viewer U1. Here, if the number of viewers U1 is small, a viewer avatar image (viewer agent) may be generated and output to the monitor 15. This viewer avatar image is displayed in the same position as the viewer U1. Thus, for example, it is displayed facing the exhibit E1, just like the viewer U1.

The output control unit 213 then determines the agent's behavior according to the consciousness state of each viewer U1. Here, it determines whether the agent can speak to each viewer U1 based on the consciousness state of each viewer U1. For example, if it is determined that viewer U1 is considering something based on the viewer's facial expressions and gestures, the agent puts the conversation on hold. Then, when the consideration ends or a predetermined waiting time has elapsed, the output control unit 213 uses voice synthesis to speak to the viewer U1 from the monitor 15 and to cause the agent image to move. If this is the first time the agent has spoken to the viewer U1, the agent will use voice synthesis to greet the viewer U1 and ask about his or her thoughts on the exhibit E1.

Next, the control unit 21 of the support server 20 executes a process for acquiring the reactions of the viewers (step S13). Specifically, the acquisition unit 211 of the control unit 21 acquires the captured images of each viewer U1 from the camera 10. Furthermore, the acquisition unit 211 acquires the voices uttered by each viewer U1 from the microphone 11.

Next, the control unit 21 of the support server 20 executes a keyword extraction process (step S14). Specifically, the analysis unit 212 of the control unit 21 analyzes the emotions of each viewer U1 in the facial expressions, gestures, and speech of each viewer U1 included in the captured image, and identifies keywords that express the emotions.

Furthermore, the analysis unit 212 identifies the pointing of the viewer U1 in the gesture of the viewer U1. Then, the analysis unit 212 identifies a keyword that represents a pointed area of the exhibit E1 that exists in the pointing direction, using the characteristic information of the exhibition information storage unit 22.
Furthermore, the analysis unit 212 extracts keywords by voice recognition of the speech of the viewer U1.

Next, the control unit 21 of the support server 20 executes a co-occurrence analysis process (step S15). Specifically, the analysis unit 212 of the control unit 21 assigns a user ID to the viewer U1 from whom the keywords were obtained. Next, the analysis unit 212 records each extracted keyword in the analysis information storage unit 23 in association with the exhibit ID and the user ID. The analysis unit 212 then generates a co-occurrence network by performing co-occurrence analysis using the keywords associated with this exhibit ID. In this case, the analysis unit 212 changes the display form (e.g., display color) in the co-occurrence network so that the keywords of other user IDs and the keywords of the current viewer U1 can be distinguished.

The control unit 21 of the support server 20 then executes a process for outputting the analysis results (step S16). Specifically, the analysis unit 212 of the control unit 21 outputs the co-occurrence network, which is the result of the co-occurrence analysis, to the monitor 15.

Next, the control unit 21 of the support server 20 executes a process of generating answer candidates (step S17). Specifically, the output control unit 213 of the control unit 21 generates answer candidates according to the results of the co-occurrence analysis. For example, when there are matches or differences between the keywords of other users and the keywords of the viewer U1 in the co-occurrence network, a response candidate that points out the matches or differences in the co-occurrence network is generated. In addition, a question about the characteristics of the exhibit E1 may be generated as an answer candidate for the viewer U1.
Then, the control unit 21 of the support server 20 repeats the process of determining the viewer's status (step S11) and subsequent processes.

Furthermore, in the agent operation process (step S12), depending on the viewer's situation, the control unit 21 of the support server 20 outputs response candidates according to the viewer's reaction. In this case, the output control unit 213 plays back the contents of the response candidates by voice synthesis in conjunction with the avatar image 151.

The control unit 21 of the support server 20 also periodically executes a feedback process to the creator (step S18). Specifically, the output control unit 213 of the control unit 21 acquires the analysis results for each exhibit E1 from the analysis information storage unit 23. The output control unit 213 then acquires the creator's contact information from the exhibition information storage unit 22 and transmits the analysis information to this contact information.

According to this embodiment, the following effects can be obtained.
(1) In this embodiment, the control unit 21 of the support server 20 executes a process of determining the viewer's situation (step S11). This allows the response to be adjusted according to the viewer's situation. For example, if the viewer U1 is thinking, a timely response can be made after waiting, etc. In addition, the response can be changed depending on whether there is one viewer U1 or multiple viewers U1.

(2) In this embodiment, the control unit 21 of the support server 20 executes an agent operation process according to the viewer's situation (step S12). This allows conversation to be conducted according to the viewer U1's situation.

(3) In this embodiment, the control unit 21 of the support server 20 executes a process for acquiring the viewers' reactions (step S13). This makes it possible to acquire keywords expressing impressions of the exhibit E1 according to the reactions of each viewer U1 of the exhibit E1.

(4) In this embodiment, the control unit 21 of the support server 20 executes a keyword extraction process (step S14). This allows the impressions of each viewer U1 to be expressed using keywords.

(5) In this embodiment, the control unit 21 of the support server 20 executes a co-occurrence analysis process (step S15). This makes it possible to express the impressions of each viewer U1 regarding the exhibit E1 by a co-occurrence network using the keywords of each viewer U1 who viewed the exhibit E1.

(6) In this embodiment, the control unit 21 of the support server 20 executes a process for outputting the analysis results (step S16). This makes it possible to encourage appreciation from a variety of perspectives by using a co-occurrence network that includes the keywords of other viewers.

(7) In this embodiment, the control unit 21 of the support server 20 executes a process for generating response candidates (step S17). This allows the viewer U1 to be prompted to use a variety of viewing methods.

(8) In this embodiment, the control unit 21 of the support server 20 periodically executes a feedback process to the creator (step S18). This allows the creator to receive feedback on information regarding the impression of the viewer U1.

This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction.
In the above embodiment, the exhibition information storage unit 22 records exhibition management information related to the exhibits exhibited at the venue. This exhibition management information includes data related to characteristic information. The characteristic information includes the characteristics (pattern and color) of the exhibit E1 relative to the exhibition position of this exhibit. The characteristic information is not limited to pattern and color as long as it can express the characteristics of the exhibit E1. Abstract expressions representing each area of the exhibit E1, keywords evoked by each area, etc. may also be used.

In the above embodiment, the monitor 15 is placed in the vicinity of the exhibit E1 in correspondence with it. Alternatively, a user terminal used by the viewer U1 may be used as the monitor 15. In this case, for example, an agent is output to the user terminal by augmented reality using the exhibit or the like as a marker.
In the above embodiment, the exhibits are described as exhibits at a venue such as an art museum. The exhibits are not limited to art pieces. For example, audio exhibits such as music may be used as the exhibits. Furthermore, texts such as novels may be used as the exhibits. Furthermore, the exhibit E1 may be placed in a virtual space.

In the above embodiment, the control unit 21 of the support server 20 executes the co-occurrence analysis process (step S15). The analysis method is not limited to the co-occurrence analysis.
In the above embodiment, the control unit 21 of the support server 20 executes a co-occurrence analysis process (step S15). Here, the co-occurrence analysis may be performed according to the attributes (gender, age group, etc.) of the viewer U1, and the analysis results may be output. In this case, analysis information of other viewers who have attributes in common with the viewer U1 is used. Furthermore, when multiple viewers U1 are detected, statistical values of the attributes of the viewer U1 (for example, the attributes of the majority) may be used.

E1...exhibit, U1...viewer, 10...camera, 11...microphone, 15...monitor, 20...support server, 21...control unit, 211...acquisition unit, 212...analysis unit, 213...output control unit, 22...exhibition information storage unit, 23...analysis information storage unit.

Claims (9)

An appreciation support system including an output device capable of providing information to a plurality of people and a control unit connected to an information acquisition device,
The control unit:
The number of viewers of the exhibit and their reactions are acquired from the information acquisition device;
Identifying the positions of agents according to the number of agents;
An appreciation support system characterized by determining the agent's response according to the reaction based on characteristic information extracted from the exhibit, and outputting the response to the output device in the orientation of the agent at the position . The control unit:
Acquire a photographed image of the viewer;
2. The appreciation support system according to claim 1, wherein the response is determined in accordance with the captured image. The viewing support system according to claim 2, characterized in that the control unit recognizes a gesture of the viewer in the captured image and determines the response in response to the gesture. The viewing support system according to claim 2, characterized in that the control unit specifies a viewer to be spoken to among multiple viewers in the captured image and outputs the response. The viewing support system according to claim 2, characterized in that the control unit determines the output timing of the response according to the captured image. The control unit:
acquiring an utterance of the viewer and performing speech recognition on the utterance to generate an analysis result;
The appreciation support system according to claim 1, further comprising: an output section for outputting the analysis results. The viewing support system according to claim 6, characterized in that the control unit determines the agent's response according to the analysis results. A viewing support method for supporting viewing by using a viewing support system including an output device capable of providing information to a plurality of people and a control unit connected to an information acquisition device, comprising:
The control unit:
The number of viewers of the exhibit and their reactions are acquired from the information acquisition device;
Identifying the positions of agents according to the number of agents;
A viewing support method characterized by determining the agent's response according to the reaction based on characteristic information extracted from the exhibit, and outputting the response to the output device in the orientation of the agent at the position . A viewing support program for supporting viewing using a viewing support system including an output device capable of providing information to a plurality of people and a control unit connected to an information acquisition device,
The control unit,
The number of viewers of the exhibit and their reactions are acquired from the information acquisition device;
Identifying the positions of agents according to the number of agents;
A viewing support program characterized by determining the agent's response according to the reaction based on characteristic information extracted from the exhibit, and having the output device function as a means for outputting the response in the orientation of the agent at the position .