JP7845690B2 - Question creation device, question creation system, and question creation method - Google Patents

Description

This invention relates to a question creation device, a question creation system, and a question creation method.

In recent years, it has become common to view videos on the user's device (e.g., tablet devices, smartphones, personal computers, etc.). For example, Patent Documents 1-3 describe viewing broadcast programs such as dramas on the viewer's device.

Japanese Patent Publication No. 2012-119833 Japanese Patent Publication No. 2007-306399 Japanese Patent Publication No. 2004-23425

Incidentally, when viewers of a video become interested in an object shown in the video (hereinafter sometimes referred to as "item"), they may try to collect information related to that item (hereinafter sometimes referred to as "item-related information"). For example, when a viewer is watching a drama, they might want to buy the bag that the main character is carrying and try to find the brand name and product name of that bag online.

Furthermore, viewers may attempt to gather item-related information using, for example, online chatbots. In recent years, many chatbots utilize software built on Large Language Models (LLMs). Viewers can obtain item-related information through text and voice interactions with the chatbot.

However, even if viewers try to obtain item-related information through text or voice interaction, it will be difficult to find the desired information if they don't know the search keywords for the item. In other words, because viewers don't know the search keywords for the item, they cannot easily come up with appropriate questions to ask the chatbot.

One example of the object of the present invention is to facilitate the creation of questions for obtaining information related to objects displayed in a video. Other objects of the present invention will become apparent from the description herein.

One aspect of the present invention is a question creation device comprising: an element output unit that extracts video elements from a video on which an action has been performed and outputs them to an analysis device when an action has been performed on the displayed video; a text acquisition unit that acquires text elements created by the analysis device based on the video elements; and a question creation unit that creates candidate questions related to the video elements from the text elements.

One aspect of the present invention is a question creation system comprising: an element output unit that extracts video elements from a video on which an action has been performed and outputs them to an analysis device when an action has been performed on the displayed video; a text acquisition unit that acquires text elements created by the analysis device based on the video elements; a question creation unit that creates candidate questions related to the video elements from the text elements; and a display unit that displays the candidate questions.

One aspect of the present invention is a question creation method comprising: extracting video elements from a displayed video when an action is performed on the video, outputting them to an analysis device; obtaining text elements created by the analysis device based on the video elements; and creating candidate questions related to the video elements from the text elements.

According to the above-described aspect of the present invention, questions can be easily created to obtain information related to the object displayed in the video.

Figure 1 is an explanatory diagram showing an example of an action performed on a video in this embodiment. Figure 2 shows a question creation system 100 having the question creation device 10 of this embodiment. Figure 3 is a flowchart of the question creation process performed by the question creation device 10 of this embodiment. Figure 4 shows an example of a question template. Figure 5 shows how to input text elements into a question template when there are multiple text elements. Figure 6 shows an example of the screen in the first example of the question creation process. Figure 7 shows an example of the screen used when acquiring video elements and creating candidate questions in the first example of the question creation process. Figure 8 shows an example of a screen displaying candidate questions in the first example of the question creation process. Figure 9 shows an example of the screen in the second example of the question creation process. Figure 10 shows an example of the screen when acquiring video elements in the second example of the question creation process. Figure 11 shows an example of the screen used when obtaining text elements and creating question candidates in the second example of the question creation process.

From the description in the specification and drawings described below, at least the following matters become clear:

A question creation device is revealed that comprises: an element output unit that extracts video elements from a displayed video when an action is performed on the video and outputs them to an analysis device; a text acquisition unit that acquires text elements created by the analysis device based on the video elements; and a question creation unit that creates candidate questions related to the video elements from the text elements. Such a question creation device makes it easy to create questions to obtain information related to objects displayed in a video.

The aforementioned video element preferably contains at least one of the following: image information, audio information, and text information related to the video. This makes it easy to create questions to obtain information related to the subject shown in the video.

The aforementioned video element has image information that includes type information related to the video element. The text acquisition unit acquires the type information linked to the text element, and the question creation unit preferably creates the question candidates using a question template corresponding to the type information. This allows for the creation of highly accurate question candidates based on the type information by utilizing the type information contained in the image information.

The aforementioned video element preferably consists of at least one of image information, audio information, and text information that does not include type information related to the video element. The text acquisition unit acquires the text elements associated with the type information, and the question creation unit creates the candidate questions using a question template corresponding to the type information. This allows the use of a question template corresponding to type information even when the video element does not have type information.

The text acquisition unit acquires a first text element associated with the first type information and a second text element associated with the second type information. The question template has a first input element related to the first type information and a second input element related to the second type information. It is desirable that the question creation unit creates a question candidate in which the first text element is input to the first input element and the second text element is input to the second input element. This prevents the question creation unit from creating question candidates with unnatural phrasing.

It is desirable to include a question output unit that outputs the aforementioned question candidates to a display device. This allows viewers to visually view the question candidates via the display device.

The question generation unit should create a plurality of candidate questions, and the question output unit should preferably output the plurality of candidate questions to the display device in a manner that allows the viewer of the video to select one. This allows the viewer to select a question that best suits their intentions from among the multiple candidate questions.

It is desirable that the question output unit adds priority information to the multiple question candidates and outputs them to the display device. This allows viewers to quickly select the question candidate that best suits their intentions from among the multiple candidates.

The question output unit should preferably output the candidate questions confirmed by the video viewer to the answer generation device. This allows for the automatic generation of answers to the questions confirmed by the viewer.

The aforementioned answer generation device should preferably be a large-scale language model. This allows for the automatic generation of answers to questions confirmed by the audience.

The aforementioned actions should preferably be performed by clicking, touching, voice input, or keyboard input, or a combination of these operations. This makes it easy to create questions to obtain information related to the subject shown in the video.

A question creation system is revealed comprising: an element output unit that extracts video elements from a video on which an action has been performed and outputs them to an analysis device when an action is performed on the displayed video; a text acquisition unit that acquires text elements created by the analysis device based on the video elements; a question creation unit that creates candidate questions related to the video elements from the text elements; and a display unit that displays the candidate questions. Such a question creation system makes it easy to create questions to obtain information related to objects shown in a video.

A question generation method is revealed, comprising: extracting video elements from a displayed video when an action is performed on the video, outputting them to an analysis device; obtaining text elements created by the analysis device based on the video elements; and creating candidate questions related to the video elements from the text elements. Such a question generation method makes it easy to create questions to obtain information related to objects shown in the video.

The following describes preferred embodiments of the present invention with reference to the drawings. The same or equivalent components, members, etc., shown in each drawing are denoted by the same reference numerals, and redundant descriptions are omitted where appropriate.

===This Embodiment===
<Video Summary>
Before describing the question creation device, question creation system, and question creation method of this embodiment, we will first describe an example of a video used in this embodiment and the actions performed on the video, with reference to Figure 1.

Figure 1 is an explanatory diagram showing an example of an action performed on a video in this embodiment. Figure 1A shows an example of a screen in the video of this embodiment, Figure 1B shows an example of an operation to store item area information in the video (described later), and Figure 1C shows an example of a screen in a state where item area information has been stored through the stocking process.

As shown in Figure 1A, the video is being played on the screen (in this case, the touch panel 6) of the user terminal 5 (for example, the viewer's smartphone or tablet). "Video" refers to moving images delivered as data from the distribution entity (in this case, the video distribution server 1, described later), such as variety shows, dramas, news, sports, music, and anime. The video distribution format is not limited to streaming, as will be described later; it may also be download, progressive download, or live.

Figure 1A shows a touch panel 6 displaying an image of a person wearing sportswear (specifically, a jacket) on their upper body. In this embodiment, since item areas are pre-defined in the frames (still images) included in the video, the video contains information about the items for which the item areas are defined (hereinafter sometimes referred to as "item area information").

Here, "item area" refers to the area on the screen corresponding to an item displayed in the video. In this embodiment, by setting an item area for an item displayed in the video and guiding the viewer to that item area information, item area information within the video can be provided to the viewer in a simple manner. For example, when a viewer becomes interested in an item in the video, they can simply select the item area to display the item area information associated with that area. In this embodiment, an item area is pre-set in the area of the jacket on the screen. In Figure 1B, the item area is represented by a rectangular dashed frame 41.

However, the frame 41 representing the item area is not displayed on the actual touch panel 6. This prevents the frame 41 from interfering with video viewing. In other words, Figure 1B shows the frame 41 indicating the item area for illustrative purposes. However, the frame 41 representing the item area may be displayed on the actual touch panel 6 during video playback, rather than being hidden.

During video playback, frames (still images) are displayed sequentially, so the area occupied by the album art on the video screen (on the frame) changes constantly. However, in this embodiment, the item area is also set to change constantly in accordance with the video, and the frame 41 representing the item area also deforms constantly.

As shown in Figure 1B, when the user terminal 5 detects that an item area pre-set in the video has been tapped, it stores the item area information associated with that item area as stock information. Here, "tap operation" refers to the action of touching and releasing the screen with a finger, and is included in "touch operation." Furthermore, "touch operation" is a general term for operations performed by touching the screen with a finger. "Touch operation" includes not only the aforementioned "tap operation," but also various operations such as "swipe," "double tap," "flick," "scroll," "drag," "long press," and "pinch in or pinch out." The user terminal 5 may also store the item area information associated with other touch operations besides tap operations as stock information.

As shown in Figure 1C, when predetermined item area information is stored as stock information on the user terminal 5, an item image 44 (e.g., a thumbnail image) corresponding to the stock information (stocked item area information) is displayed on the stock information display unit 43 on the touch panel 6. That is, when a viewer becomes interested in an album cover in a video, they can tap the cover on the touch panel 6 to stock the item area information for that cover. Furthermore, the viewer can confirm on the stock information display unit 43 that the item area information for that cover has been stored as stock information.

Although not shown in Figures 1B and 1C, when a viewer stores item area information, the item image (e.g., a thumbnail image) associated with that item area may be displayed. For example, when the user terminal 5 detects that an item area pre-set in the video has been tapped, it displays the item image associated with that item area. Even if the frame 41 indicating the item area (the dashed rectangle in Figure 1B) is hidden, the viewer can recognize that by tapping the jacket on the touch panel 6 when interested in the jacket in the video, the item image of the jacket will be displayed, allowing them to obtain related information about that jacket. Furthermore, the user terminal 5 may also display the item image associated with an item area when it detects other touch operations, such as a double tap, instead of a single tap. Additionally, the viewer may store information for multiple item areas.

When the user terminal 5 detects that an item image 44 displayed on the stock information display unit 43 has been tapped, it performs processing according to the event information associated with that item image 44 (item area information) (for example, displaying the jacket purchase page (payment page)). The user terminal 5 may also perform processing according to the event information associated with the item image 44 (item area information) if it detects any touch operation other than a tap.

Here, the item area information for the jacket is associated with the address of the jacket's purchase page (payment page). When the viewer taps the jacket's item image 44, the jacket's purchase page (payment page) is displayed on the touch panel 6. The display screen for the jacket's purchase page (payment page) is both the item area information for the jacket and the jacket's item image 44. The webpage can be displayed as part of a multi-screen setup alongside the currently playing video, or it can be displayed independently.

In the following explanation, actions taken by viewers in response to the video they are watching may be referred to as "actions." Actions are not limited to the touch operations (taps and swipes) described above. Actions taken by viewers in response to the video they are watching may include, for example, voice input, or, if the user's device 5 is a personal computer, mouse clicks or keyboard input. Actions are performed by clicking, touching, voice input, keyboard input, or a combination of any of these operations.

Furthermore, in the example described above, the terminal (device) on which the action is performed is the same terminal (device) on which the video is displayed. That is, on user terminal 5, where the video is displayed, the storage of stock information via tap operations and the display of the item purchase page (payment page) via tap operations are performed.

However, the terminal (device) on which the action is performed may be different from the terminal (device) on which the video is displayed. The action may be performed on a terminal other than the user terminal 5 on which the video is displayed, for example, on a different smartphone.

Furthermore, the frames (still images) included in the video do not need to have item areas defined. In other words, the video does not need to contain item area information.

<Question Creation System 100>
Next, with reference to Figure 2, the configuration of the question creation system 100 having the question creation device 10 of this embodiment will be described.

Figure 2 shows a question creation system 100 having the question creation device 10 of this embodiment.

The question creation system 100 is a system for creating questions to obtain item-related information. This item-related information includes not only information directly related to the item (e.g., item name, size, color, price, etc., and item area information in Figure 1 above), but also indirect information about the item (e.g., recommended information related to the item).

In this embodiment, when a viewer becomes interested in an object (item) displayed in a video, they can easily create a question to obtain item-related information simply by performing an action on the displayed video. For example, if a viewer were to input text on their user terminal when creating a question for a chatbot, it would interfere with video viewing. In contrast, this embodiment allows for the automatic creation of questions to obtain item-related information simply by performing an action on the displayed video, thus minimizing interference with video viewing. In the question creation system 100, the question creation process is executed by multiple devices via the communication network 9 described later.

The question creation system 100, as shown in Figure 2, comprises a video distribution server 1, a metadata distribution server 3, a user terminal 5, a question creation device 10, an analysis device 20, and an answer creation device 30. The video distribution server 1, metadata distribution server 3, user terminal 5, question creation device 10, analysis device 20, and answer creation device 30 are interconnected and can communicate with each other via a communication network 9. Here, the communication network 9 could be, for example, the Internet, a telephone network, a wireless communication network, a LAN, or a VAN; in this case, the Internet is assumed.

The video distribution server 1 is a server for distributing video. In this embodiment, the video distribution server 1 distributes video data to the user terminal 5 in streaming format. However, the video data distribution method may also be download format or progressive download format. In the case of streaming format distribution, the user terminal 5 temporarily stores the video data, and in the case of download format distribution, the user terminal 5 stores and retains the downloaded video data. Furthermore, the video distribution server 1 may also distribute video in live format.

The metadata distribution server 3 is a server for distributing metadata, including the item area information described above. In this embodiment, a portion of the metadata is distributed in a preload format before video playback, while another portion of the metadata is distributed in a progressive download format. However, the method of metadata distribution is not limited to this; for example, it may be in download format or streaming format.

In this embodiment, for the sake of explanation, metadata and video data are described separately, but metadata may be stored within the video data (video file). If the video distribution server 1 distributes video data with metadata stored within it, the question creation system 100 does not need to have a metadata distribution server 3.

Furthermore, the metadata distributed by the metadata distribution server 3 may be created by the metadata distribution server 3, or it may be created by a metadata creation terminal (not shown).

User terminal 5 is an information terminal (video playback device) capable of playing videos. Here, user terminal 5 is a smartphone. However, user terminal 5 is not limited to a smartphone; for example, it could be a tablet-type mobile device or a personal computer. User terminal 5 is equipped with hardware such as a CPU, memory, storage device, communication module, and touch panel 6 (corresponding to display unit 7A and input unit 7B), which are not shown. A video playback program is installed on user terminal 5, and the various operations described above are realized when user terminal 5 executes this video playback program. The video playback program can be downloaded to user terminal 5 from a program distribution server (not shown).

The user terminal 5 comprises a display unit 7A, an input unit 7B, a control unit 8A, and a communication unit 8B.

The display unit 7A is responsible for displaying various screens. In this embodiment, the display unit 7A is implemented by the display of the touch panel 6 and a controller that controls the display of that display. The input unit 7B is responsible for inputting and detecting user instructions. In this embodiment, the input unit 7B is implemented by the touch sensor of the touch panel 6. Note that if the user terminal 5 is a smartphone or a tablet-type mobile terminal, the display unit 7A and the input unit 7B are mainly implemented by the touch panel 6, but the display unit 7A and the input unit 7B may be composed of separate components. For example, if the user terminal 5 is a personal computer, the display unit 7A would be composed of, for example, a liquid crystal display, and the input unit 7B would be composed of a mouse or keyboard.

The control unit 8A controls the user terminal 5. The control unit 8A has functions for processing video data and playing (displaying) the video, as well as functions for processing metadata. Furthermore, the control unit 8A also has browser functions for acquiring web page information and displaying web pages. In this embodiment, the control unit 8A is implemented by a CPU (not shown), memory and storage devices that store the video playback program, etc.

The communication unit 8B is responsible for connecting to the communication network 9. The communication unit 8B receives video data from the video distribution server 1, receives metadata from the metadata distribution server 3, and requests data from the video distribution server 1 and the metadata distribution server 3.

Although not shown, the user terminal 5 may also have, in addition to the configuration described above, a video data storage unit with the function of storing video data, a metadata storage unit with the function of storing metadata, and a stock information storage unit that stores stocked item area information in association with video data.

The question creation device 10 is a device for creating questions to obtain item-related information. The question creation device 10 executes the question creation process described later. In this embodiment, the execution of various processes, including the question creation process, becomes possible through the cooperation of various hardware and software (not shown) of the question creation device 10.

The question generation device 10 is, for example, a computer such as a server, and is composed of an arithmetic unit (CPU, etc.), memory, storage device, communication device, etc. The storage device stores various programs, including the question generation program, and various data. The arithmetic unit reads the question generation program stored in the storage device into memory and executes it, thereby realizing various processes, including the question generation process, i.e., the functions of each part described later (element output unit 11, text acquisition unit 12, question generation unit 13, and question output unit 14). Details of the functions of the element output unit 11, text acquisition unit 12, question generation unit 13, and question output unit 14 will be described later.

The question creation device 10 in this embodiment may have multiple computers. Furthermore, various processes, including the question creation process, may be executed through the cooperation of these multiple computers via a network. In the question creation system 100 shown in Figure 2, the question creation device 10 is connected to one user terminal 5 via a communication network 9, but the question creation device 10 may be connected to multiple user terminals 5 via the communication network 9.

The analysis device 20 is a device that analyzes video elements in a video and creates text elements.

Here, a video element is an element that can be extracted as data from a video. A video element, for example, has at least one of the following elements related to the video: image information, audio information, and text information. Note that image information includes, for example, image data of people or objects in the video, and is not limited to image data that includes item area information (i.e., image data for which an item area has been set), but also includes image data that does not include item area information (i.e., image data for which an item area has not been set). Audio information is audio data such as sounds, dialogue, and background music in the video. Text information is not limited to text information within the video, such as subtitles and captions, but also includes text information outside the video, such as viewer comments.

Furthermore, a text element is textual information that represents the video element mentioned above. For example, if the video element is image data of a jacket as shown in Figure 1 above, the textual information would be "clothing," "sportswear," "jacket," etc. If the jacket is red, the textual information would be "red," etc. Note that there is no one-to-one correspondence between video elements and text elements; a single video element may be associated with multiple text elements.

In this embodiment, the analysis device 20 can perform various processes, including analysis processing, through the cooperation of various hardware (not shown) and various software.

The analysis device 20 is, for example, a computer such as a server, and is composed of a processing unit (CPU, etc.), memory, storage device, communication device, etc. The storage device stores various programs, including the analysis program, and various data. The processing unit reads the analysis program stored in the storage device into memory and executes it, thereby realizing various processes, including the analysis process. The analysis device 20 in this embodiment may have multiple computers. Furthermore, various processes, including the analysis process, may be executed through the cooperation of these multiple computers via a network.

The analysis device 20 specifically includes large-scale language models, text search engines, image search/generation engines, sound source search/generation engines, video search/generation engines, recommendation engines, etc. The analysis device 20 is not limited to devices using AI (Artificial Intelligence); any device that analyzes video elements in a video and creates text elements, even if it does not use AI, is acceptable.

In the analysis device 20, for example, the text search engine and the image search engine may be configured on separate computers. Furthermore, the computer constituting the text search engine may be composed of multiple computers.

In this embodiment, the analysis device 20 was described as a separate device from the question creation device 10. That is, the analysis device 20 was described as an external device to the question creation device 10. However, the question creation device 10 may also possess the functions of the analysis device 20. Furthermore, the question creation system 100 does not necessarily include the analysis device 20.

The answer generation device 30 is a device that generates answers to candidate questions related to video elements created from the text elements described above. In this embodiment, the cooperation of various hardware and software (not shown) of the answer generation device 30 enables the execution of various processes, including the answer generation process.

The answer generation device 30 is, for example, a computer such as a server, and is composed of an arithmetic unit (CPU, etc.), memory, storage device, communication device, etc. The storage device stores various programs, including the answer generation program, and various data. The arithmetic unit reads the answer generation program stored in the storage device into memory and executes it, thereby realizing various processes, including the answer generation process. The answer generation device 30 in this embodiment may have multiple computers. Furthermore, various processes, including the answer generation process, may be executed through the cooperation of these multiple computers via a network.

The answer generation device 30 specifically includes a large-scale language model, a text search engine, an image search/image generation engine, an audio search/audio generation engine, a video search/video generation engine, a recommendation engine, etc. In this embodiment, specifically, the answer generation device 30 is a generative AI built on a large-scale language model such as CHATGPT®. The answer generation device 30 is not limited to an AI-based device; any device that generates answers to candidate questions may also be an AI-based device.

In this embodiment, the answer generation device 30 was described as a separate device from the question generation device 10. That is, the answer generation device 30 was described as an external device to the question generation device 10. However, the question generation device 10 may also possess the functions of the answer generation device 30. Furthermore, the question generation system 100 does not necessarily include the answer generation device 30.

<Question generation device 10>
As shown in Figure 2, the question creation device 10 includes an element output unit 11, a text acquisition unit 12, a question creation unit 13, a question output unit 14, and a recording unit 16.

The element output unit 11 is the part that extracts video elements (e.g., image information of the jacket) from the displayed video when an action is performed on the video, and outputs them to the analysis device 20. As described above, the analysis device 20 analyzes the video elements (e.g., image information of the jacket) output from the element output unit 11 and creates text elements (e.g., the text information "jacket").

Furthermore, the analysis device 20 may create multiple text elements (for example, text information such as "jacket," "clothing," and "sportswear") for a single video element (for example, image information of a jacket). The analysis device 20 creates text elements (for example, text information such as "jacket") based on the video element (for example, image information of a jacket) output from the element output unit 11, and outputs them to the text acquisition unit 12.

The text acquisition unit 12 is the part that acquires at least one text element created based on the video elements by the analysis device 20.

The question generation unit 13 is responsible for creating candidate questions (for example, "How much does the jacket cost?") related to video elements from the text elements (for example, the text information "jacket") acquired by the text acquisition unit 12. The question generation unit 13 may also create multiple candidate questions (for example, "How much does the jacket cost?", "Where is the jacket sold?") from a single text element (for example, the text information "jacket").

The question output unit 14 is the part that outputs the question candidates created by the question creation unit 13 to a display device (for example, the display unit 7A of the user terminal 5). This allows viewers to visually view the question candidates via the display device.

The recording unit 16 is the part where the video elements, text elements, and question candidate data described above are recorded. However, the recording unit 16 may also record data other than the video elements, text elements, and question candidate data. Furthermore, the question creation device 10 does not necessarily have a recording unit 16; the user terminal 5 may have a recording unit 16. Specifically, the user terminal 5 may have a recording unit 16 as a memory area. This allows specific video elements, text elements, and question candidates to be recorded on the user terminal 5 (i.e., the terminal owned by the viewer) while protecting the viewer's privacy. Moreover, both the question creation device 10 and the user terminal 5 may have a recording unit 16.

<Overview of the question creation process>
Next, with reference to Figure 3, we will describe a question creation process that is an example of the question creation method of this embodiment.

Figure 3 is a flowchart of the question creation process performed by the question creation device 10 of this embodiment.

First, when an action is performed on the displayed video, the element output unit 11 extracts video elements (e.g., image information of the jacket) from the video in which the action was performed and outputs them to the analysis device 20 (S001). Here, the action is not limited to one; multiple actions may be performed, and the element output unit 11 may extract at least one video element between multiple actions. For example, the element output unit 11 may extract video elements during the period when the viewer performs multiple touch operations within a predetermined time. This allows the element output unit 11 to extract not only video elements at the time the action was performed, but also video elements within a certain time range (a predetermined time range). Furthermore, the element output unit 11 may extract video elements during the period when the viewer performs a long press operation within a predetermined time.

The analysis device 20 creates text elements (for example, the text information "jacket") based on the video elements (for example, image information of the jacket) output from the element output unit 11, and outputs them to the text acquisition unit 12.

Next, the text acquisition unit 12 acquires text elements (for example, the text information "jacket") from the analysis device 20 (S002). Then, the question creation unit 13 creates a question candidate (for example, the text information "How much does the jacket cost?") (S003).

In this embodiment, the question creation unit 13 creates question candidates using a question template corresponding to the type information. Here, "type information" refers to textual information related to the attributes of a video element, and is data that can be linked to the text element of that video element. For example, if the video element is image data of a jacket, the text element of that video element is the textual information "jacket," and the type information is the textual information "proper name/product name."

If a video element is image information that includes item area information (i.e., image data of a video as shown in Figure 1), then the type information may be included in the item area information of that video element. That is, the type information may be included as part of the video's item area information. In this case, the text acquisition unit 12 acquires the type information (for example, the text information "proper name/product name") by associating it with a text element (for example, the text information "jacket"). By utilizing the type information contained in the image information, it is possible to create highly accurate question candidates based on the type information.

If a video element consists of at least one of the following: image information, audio information, or text information, without item area information, the analysis device 20 may analyze the video element and create a text element (e.g., text information such as "jacket") associated with type information (e.g., text information such as "proper name/product name"). The text acquisition unit 12 acquires the text element associated with the type information. This allows the use of a question template corresponding to type information even if the video element does not have type information.

The question templates are pre-recorded in the recording unit 16, and the question creation unit 13 reads the question template data from the recording unit 16 to create candidate questions.

Figure 4 shows an example of a question template.

As shown in Figure 4, question templates are provided, such as "I want to know more about ○○○○○," "How much does ○○○○○ cost?", "Where is ○○○○○ sold?", and "I want to see different colors of ○○○○○." Here, the "○○○○○" part of the question template is where text elements (for example, the text information "jacket") are entered (hereinafter sometimes referred to as "input elements"). Furthermore, the four question templates mentioned above correspond to category information, which is "proper nouns/product names."

In the case of text elements (e.g., "jacket") associated with type information (e.g., text information such as "proper name/product name"), the question creation unit 13 uses the question template corresponding to the type information "proper name/product name" from among multiple question templates. By using a question template corresponding to the type information, rather than arbitrarily selecting and using a question template (unrelated to the type information), the question creation unit 13 can create more accurate question candidates that better align with the viewer's intent.

Figure 5 shows how to input text elements into a question template when there are multiple text elements.

The question template is not limited to having only one input element; as shown in Figure 5A, it may have multiple input elements (in this case, a first input element and a second input element). Furthermore, each input element may be linked to a predetermined type of information. That is, the first input element (○○○) may be linked to a first type of information (e.g., "person's name"), and the second input element (△△△) may be linked to a second type of information (e.g., "proper name/product name").

Here, if the text elements entered into the first input element and the text elements entered into the second input element are arbitrary (unrelated to type information), then, for example, a candidate question like the one shown in Figure 5B may be generated. In other words, a candidate question with an unnatural expression, such as "I want to know more about Mr. ABC, who was wearing the jacket," may be generated.

In this embodiment, since each input element is linked to predetermined type information, the question creation unit 13 creates a candidate question as shown in Figure 5C, where the first text element (Mr. ABC) is input to the first input element (○○○) and the second text element (Jacket) is input to the second input element (△△△). This makes it possible to create a candidate question that is expressed naturally, such as "I would like to know more about the jacket Mr. ABC was wearing." In other words, it is possible to suppress the creation of a candidate question that is expressed unnaturally, as shown in Figure 5B.

Next, the question output unit 14 outputs the question candidates to the display device (S003). This allows viewers to visually view the question candidates via the display device. Furthermore, in this embodiment, since the question candidates are displayed during video playback (in parallel with video playback), viewers can review the question candidates without interrupting their video viewing.

When the question generation unit 13 generates multiple question candidates, the question output unit 14 may output the multiple question candidates to the display device in a manner that allows the video viewer to select one. This allows the viewer to select the question candidate that best suits their intentions from among the multiple candidates.

Furthermore, the question output unit 14 may output multiple question candidates to the display device with priority information added. For example, multiple question candidates may be displayed from top to bottom of the screen in order of priority. Alternatively, multiple question candidates may be displayed in order of priority through screen transitions. Additionally, each of the multiple question candidates may be displayed with relevance information (relevance percentage) and priority ranking information (1st, 2nd, 3rd, etc.). This allows viewers to quickly select the question candidate that best suits their intent from among multiple candidates.

Finally, the question output unit 14 outputs the question candidates selected by the viewer (hereinafter sometimes simply referred to as "questions") to the answer creation device 30 (S004). The answer creation device 30 creates an answer to the question and outputs it to the question creation device 10. The question creation device 10 outputs the answer to the question to a display device (for example, the display unit 7A of the user terminal 5). This allows the viewer to visually view the answer to the question via the display device.

The question generation device 10 may also record the answers to the questions obtained from the answer generation device 30 in the recording unit 16. This allows for further analysis of the answers to the questions and their use in the distributed videos. Furthermore, viewer trends can be extracted from the analysis results, and recommendations can be created based on these trends.

<Specific examples of the question creation process>
Next, we will explain a specific example of the question creation process, referring to Figures 6 to 11.
- First Example Figure 6 shows an example of the screen in the first example of the question creation process. Figure 7 shows an example of the screen when acquiring video elements and creating question candidates in the first example of the question creation process. Figure 8 shows an example of the screen where question candidates are displayed in the first example of the question creation process.

The first example of the question generation process shown in Figures 6 to 8 is an example where, when a viewer becomes interested in an item with an item area set (i.e., a jacket), a question is generated to obtain item-related information (information about the jacket) related to that item.

As shown in Figure 6, a question button 50 is located in the lower right corner of the touch panel 6 of the user terminal 5. This allows viewers to recognize the existence of a function that generates question candidates. As shown in Figure 7, when the user terminal 5 detects that an item area pre-set in the video has been tapped, the question creation device 10 generates question candidates using its respective components (functions): the element output unit 11, the text acquisition unit 12, and the question creation unit 13.

In this embodiment, simply performing an action (in this case, a tap) on the displayed video automatically generates questions to obtain item-related information about the jacket, thus preventing interruptions to video viewing. As described above, when the user terminal 5 detects that an item area pre-set in the video has been tapped, it stores the item area information associated with that item area as stock information. Therefore, a single action (in this case, a tap) on the displayed video can simultaneously generate questions to obtain item-related information and store it as stock information.

In Figure 7, although a list of potential questions has been created, the specific content of the questions is not displayed on the touch panel 6. However, the viewer can recognize that a list of potential questions has been created by the change in color of the question button 50 located in the lower right corner of the touch panel 6.

As shown in Figure 8, when the user terminal 5 detects that the question button 50 has been tapped, the question output unit 14 of the question creation device 10 displays the candidate questions 52. Since the viewer can choose when to display the candidate questions 52 (the timing of tapping the question button 50), the unexpected display of candidate questions 52 can prevent interruptions to video viewing. In Figure 8, the question output unit 14 outputs multiple (in this case, three) candidate questions to the touch panel 6 in a format that the viewer can select. This allows the viewer to choose the question that best suits their intentions from among the multiple candidate questions.

・Second Example Figure 9 shows an example of the screen in the second example of the question creation process. Figure 10 shows an example of the screen when acquiring video elements in the second example of the question creation process. Figure 11 shows an example of the screen when acquiring text elements and creating question candidates in the second example of the question creation process.

The second example of the question generation process shown in Figures 9 to 11 is an example where, when a viewer becomes interested in an item other than the item for which an item area is set (i.e., a person), a question is generated to obtain item-related information (information about the jacket) for that item.

As shown in Figure 9, when the user terminal 5 detects that a tap operation has been performed on an area other than the item areas pre-set in the video, the element output unit 11 of the question creation device 10 outputs image information of a predetermined size area including the tapped area to the analysis device 20. As shown in Figure 10, when the viewer taps the image information of the predetermined size area, the item image within that area may be displayed. The viewer can recognize that even if the area is outside the pre-set item areas in the video, tapping a person on the touch panel 6 when they are interested in a person in the video will display the person's item image, allowing them to obtain item-related information about that person.

The analysis device 20 analyzes the image information and creates text elements. As shown in Figure 10, the text elements ("ABC") are displayed on the touch panel 6, allowing the user to recognize that the text elements were created based on the image information of the tapped area. Furthermore, as shown in Figure 11, the question creation unit 13 of the question creation device 10 creates a candidate question 52, and the question output unit 14 displays the candidate question 52.

Other Examples The examples of the question creation process are not limited to the first and second examples described above. For example, in the first example described above, the question button 50 for displaying the question candidate 52 does not need to be provided. In this case, when a viewer becomes interested in an item with an item area set (i.e., a jacket), for example, by long-pressing the item, the question candidate 52 shown in Figure 11 in the second example may be displayed. Furthermore, in the second example described above, the question button 50 may be provided, and the question candidate 52 may be displayed when the viewer taps the question button 50.

===Other===
The embodiments described above are provided to facilitate understanding of the present invention and are not intended to limit its interpretation. The present invention can be modified and improved without departing from its spirit, and it goes without saying that the present invention includes equivalents thereof.

1 Video distribution server 3 Metadata distribution server 5 User terminal 6 Touch panel 7A Display unit 7B Input unit 8A Control unit 8B Communication unit 9 Communication network 10 Question creation device 11 Element output unit 12 Text acquisition unit 13 Question creation unit 14 Question output unit 16 Recording unit 20 Analysis device 30 Answer creation device 41 Frame 43 Stock information display unit 44 Item image 50 Question button 51 Text element 52 Question candidate 100 Question creation system

Claims (13)

An element output unit that, when an action is performed on a displayed video, extracts video elements from the video on which the action was performed and outputs them to an analysis device,
A text acquisition unit that acquires text elements created based on the video elements by the analysis device,
A question creation unit that creates candidate questions related to the video elements from the text elements,
A question generation device equipped with the following features. The aforementioned video element has at least one of the following related to the video: image information, audio information, and text information.
The question generation device according to claim 1. The aforementioned video element has the image information which includes type information relating to the video element,
The text acquisition unit acquires the type information linked to the text element,
The question creation unit creates the candidate questions using a question template corresponding to the type information.
The question generation device according to claim 2. The aforementioned video element is at least one of image information, audio information, and text information that does not include type information related to the video element.
The text acquisition unit acquires the text elements associated with the type information,
The question creation unit creates the candidate questions using a question template corresponding to the type information.
The question generation device according to claim 2. The text acquisition unit acquires a first text element associated with the first type information and a second text element associated with the second type information.
The aforementioned question template includes a first input element related to the first type information and a second input element related to the second type information.
The question creation unit creates a candidate question in which the first text element is input into the first input element and the second text element is input into the second input element.
The question generation device according to claim 3 or 4. The system includes a question output unit that outputs the aforementioned candidate questions to a display device.
The question generation device according to claim 1. The question generation unit generates a plurality of candidate questions,
The question output unit outputs the plurality of question candidates to the display device in a manner that can be selected by the viewer of the video.
The question generation device according to claim 6. The question output unit adds priority information to the plurality of candidate questions and outputs them to the display device.
The question generation device according to claim 7. The question output unit outputs the candidate questions determined by the viewer of the video to the answer creation device.
A question generation device according to any one of claims 6 to 8. The aforementioned response generation device is a large-scale language model,
The question generation device according to claim 9. The aforementioned action is performed by one of the following operations: click, touch, voice input, or keyboard input, or a combination of any of these operations.
The question generation device according to claim 1. An element output unit that, when an action is performed on a displayed video, extracts video elements from the video on which the action was performed and outputs them to an analysis device,
A text acquisition unit that acquires text elements created based on the video elements by the analysis device,
A question creation unit that creates candidate questions related to the video elements from the text elements,
A display unit that displays the aforementioned candidate questions,
A question creation system equipped with the following features. When an action is performed on the displayed video, the video elements of the video on which the action was performed are extracted and output to the analysis device.
The analysis device obtains text elements created based on the video elements,
To generate candidate questions related to the video elements from the aforementioned text elements,
A question creation method that includes the following features.