JP5047882B2 - Utterance related information presentation apparatus and method - Google Patents

Description

  The present invention relates to an utterance-related information presentation apparatus and method for presenting information related to an utterance such as a meeting, and can deal with a case where each person wants to see information individually, for example, and does not hinder the discussion, and the speaker The present invention relates to an utterance related information presentation apparatus and method capable of efficiently presenting information.

  Conventionally, when presenting information such as advertisements, there is a method of presenting information according to the searcher's interest based on a search keyword on a Web page or the like. However, in the conventional method, each person inputs a keyword, and information is presented on the monitor of each person's personal computer (hereinafter referred to as a PC device) or the screen of a portable terminal. Furthermore, it does not support the use of sharing information according to each person's interest.

  On the other hand, even in the conventional method, the PC device monitor screen on which information according to each person's interest is displayed is displayed on the screen in front of the conference room, etc., so that information is presented to all conference participants. Is possible. Non-Patent Document 1 discloses a method of projecting information according to each person's uttered voice on the center of the table.

Atsuhiro Nakamo, Akira Kobayashi, Akira Nakayama, “Analysis of Remote Communication by Image Stimulation on Table Top”, Proceedings of Human Interface Symposium (CD / ROM), Vol. 2006, Page. 2422, 2006

However, the conventional method is usually not particularly problematic, but according to the inventor's study, one person can view and share one piece of information at the same time. There is room for improvement.
In addition, the method disclosed in Non-Patent Document 1 may interfere with the discussion because information is constantly presented at the center of the table. Furthermore, since the method disclosed in Non-Patent Document 1 presents information regardless of the position of the speaker, there is room for improvement in that the information is not efficiently presented to a more interested speaker. There is.

  The present invention has been made in consideration of the above circumstances, and can provide a case where each person wants to see information individually, can not deal with the discussion, and can efficiently present information to the speaker. An object is to provide an apparatus and method.

  In order to achieve the above object, a first aspect of the present invention is an utterance related information presentation device that presents utterance related information related to an utterance of a user who has uttered among a plurality of users, the utterance related information An information database in which information and search keywords are stored in association with each other, a plurality of microphones arranged at different positions, for converting the utterances into voice signals and inputting them, and an area surrounded by each user Presentation area setting means for setting presentation area data indicating a presentation area that is an outer peripheral area, and foreground area setting for setting a plurality of foreground area data indicating the near area of each user among the presentation areas On the basis of the means, the position of each microphone, and the arrival time when the utterance reached each microphone, before the user who made the utterance from the respective near area data A presentation start position calculation unit that selects area data and calculates a presentation start position from the near area data, and compares all search keywords in the information database with the utterance information corresponding to the voice signal, and the utterance Search keyword extraction means for extracting a search keyword that matches information from the information database, information extraction means for extracting utterance related information related to the extracted search keyword from the information database, and the set presentation area data And, based on the calculated presentation start position, presentation position calculation means for calculating the presentation position of the utterance-related information according to the time change so as to move from the presentation start position and circulate on the presentation area. , Based on the calculated presentation position and the extracted utterance related information, the utterance related information is A speech-related information presentation apparatus, comprising: a presentation image creating means for creating presentation image data to present the information presentation means for presenting the presentation image data, to the.

  According to a second aspect of the present invention, in the first aspect, the information database further includes position detection means arranged on the information presentation surface of the information presentation means for detecting a selection position selected by a user. Associated information is stored in association with the utterance-related information, and the presentation image creation means includes the accompanying information from the information database based on the utterance-related information presented on the detected selected position. Information is extracted, and presentation image data is created so as to present this accompanying information.

  According to a third aspect of the present invention, in the second aspect, based on the presentation area data, a central area of the areas surrounded by the presentation area is calculated, and the central area indicating the central area as the calculation result is calculated. The image processing apparatus further includes a center area calculating unit that sends area data to the presentation image creating unit. The presentation image creating unit is configured such that the selected position is directed toward the center area based on the center area data and the selected position. The presentation image data is created based on the central area data so that the utterance related information presented on the selected position is presented in the central area when moved.

(Function)
According to the first aspect, among the presentation areas that are outer peripheral areas within the area surrounded by each user, the presentation position is circulated on the presentation area from the presentation start position in the area in front of the user who has spoken. Calculation is made according to a change over time, presentation image data for presenting utterance related information related to the user's utterance at the presentation position is created, and the presentation image data is presented.

  At this time, since the utterance related information is presented so as to go around the presentation area, it is possible to cope with the case where each person wants to see the information individually, and the outer peripheral area of the area surrounded by each person is the presentation area. Since there is a presentation start position in the area in front of the user who made the utterance, information can be efficiently presented to the utterer.

  According to the second aspect, in the first aspect, the selected position selected by the user is detected, and the accompanying information related to the utterance related information is based on the utterance related information presented on the selected position. In addition to the action of the first aspect, it is possible to cope with the case where each person wants to see accompanying information individually, and to promote discussion related to the accompanying information. The accompanying information can be efficiently presented.

  According to the third aspect, in the second aspect, when the selected position selected by the user moves toward the central area, the utterance related information presented on the selected position is presented to the central area. In addition to the action of the second aspect, it is possible to respond to the case where each person wants to see information at the same time, and to promote discussion related to the information in the central area, and to present information efficiently to each person can do.

  As described above, according to the present invention, it is possible to cope with a case where each person wants to see information individually, and it is possible to efficiently present information to the speaker without disturbing the discussion.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic diagram showing an appearance of an utterance related information presentation device according to the first embodiment of the present invention, FIG. 2 is a schematic diagram showing a configuration of the device, and FIG. 3 is used in the device. It is a schematic diagram which shows the structure of an advertisement information database. This utterance related information presentation device includes three microphones 11 to 13, a PC device 20, and an information presentation unit 30.

  Here, each microphone 11-13 is arrange | positioned in a mutually different position, converts an utterance into an audio | voice signal, and inputs it into PC apparatus. As the positions of the microphones 11 to 13, for example, positions that are vertices of equilateral triangles are used in the upper space facing the substantially center of the information presentation unit 30. Note that one microphone 11 to 13 may be used for each user, but here, three microphones 11 to 13 are simultaneously used for four users. That is, the number of microphones and the number of users are not necessarily related.

  The PC device 20 includes an advertisement information database 21, a presentation area setting unit 22, a presentation start position calculation unit 23, a search keyword extraction unit 24, an information extraction unit 25, and a presentation image calculation unit 26 in addition to the functions of a normal personal computer. ing. Each unit in the PC device 20 can be realized by a CPU (not shown) executing a program installed in the PC device 20 from a network or a storage medium in advance. The same applies to the following embodiments.

  The advertisement information database 21 is a storage device that can be read / written from an input device such as a keyboard or a mouse (not shown), the search keyword extraction unit 24, the information extraction unit 25, and the like, and includes advertisement information (utterance related information) and search keywords. Are stored in association with each other. The advertisement information is an example of utterance related information. The utterance related information is not limited to advertisement information, and arbitrary information can be used. Moreover, as a situation using advertisement information, for example, in a meeting for determining a travel plan, from the utterances of each user (conference participant), advertisement information related to travel (tourist guidance, accommodation information, transportation, travel goods, Etc.) is assumed to be moved around the periphery of the table surface (information presentation surface of the information presentation unit 30) surrounded by each user like a sushi roll.

  The presentation area setting unit 22 has a function of setting presentation area data indicating a presentation area that is an outer peripheral area among areas surrounded by each user in accordance with a setting command input from an input device (not shown), and the set presentation area And a function of sending data to the presentation start position calculation unit 23 and the presentation image calculation unit 26.

The presentation start position calculation unit 23 includes a front region setting unit 23a, a region information storage unit 23b, and a front region selection unit 23c.
The foreground area setting unit 23a has a function for setting a plurality of foreground area data indicating the foreground area of each user among the presentation areas indicated in the presentation area data received from the presentation area setting unit 22, and foreground area data for each user Has a function of writing the data in the area information storage unit 23b.

The area information storage unit 23b is a storage device that stores the near area data of each user written from the near area setting unit 23a.
The foreground region selection unit 23c is configured to store each foreground region in the region information storage unit 23b based on the positions of the microphones 11 to 13 received from the search keyword extraction unit 24 and the arrival times of the utterances to the microphones 11 to 13. It has a function of selecting the near area data of the user who made the utterance from the data, calculating a presentation start position from the near area data, and a function of sending the presentation start position to the presentation image calculation unit 26.

The search keyword extraction unit 24 includes an audio processing unit 24a and a keyword comparison unit 24b.
The voice processing unit 24a presents the function of generating utterance information corresponding to the voice signal input from each microphone 11-13 and sending it to the keyword comparison unit 24b, the position of each microphone 11-13, and the arrival time of the utterance. And a function for sending to the start position calculation unit 23.

  The keyword comparison unit 24b compares all search keywords in the advertisement information database 21 with the utterance information received from the voice processing unit 24a, and extracts a search keyword that matches the utterance information from the information database. And a function of sending the retrieved keyword to the information extraction unit 25.

The information extraction unit 25 has a function of extracting advertisement information related to the search keyword transmitted from the keyword comparison unit 24 b from the advertisement information database 21 and a function of transmitting the extracted advertisement information to the presentation image calculation unit 26.
The presentation image calculation unit 26 includes a presentation position calculation unit 26a and a presentation image creation unit 26b.

  The presentation position calculation unit 26a is based on the presentation area data set by the presentation area setting unit 22, the current time received from the presentation image creation unit 26b, and the presentation start position calculated by the presentation start position calculation unit 23. It has a function of calculating the presentation position of the advertisement information according to the time change so as to move around the presentation area by moving from the presentation start position, and a function of sending the calculated presentation position to the presentation image creation unit 26b. . The presentation position calculation unit 26a may receive the current time from a timer function unit (not shown) without receiving the current time from the presentation image creation unit 26b.

  Based on the presentation position calculated by the presentation position calculation unit 26a and the advertisement information extracted by the information extraction unit 25, the presentation image creation unit 26b creates presentation image data so as to present this advertisement information at the presentation position. And a function of sending the presentation image data to the information presentation unit 30.

  The information presentation unit 30 has a function of a normal display device, is used as a table so that the information presentation surface is substantially horizontal, and has a function of presenting presentation image data received from the presentation image creation unit 26b. Here, as the display device, for example, any device such as an EL display device, a liquid crystal display device, or a plasma display device can be used. The information presentation unit 30 is not limited to the case where the display device is used as a table, but may be realized as a projector device that projects information on a normal table that is not a display device.

Next, the speech related information presentation method by the speech related information presentation device configured as described above will be described with reference to the flowchart of FIG.
First, in the PC device 20, advertisement information (utterance related information) and a search keyword are associated with each other and registered in the advertisement information database 21 by operation of an input device such as a keyboard or a mouse (not shown) (ST1). As search keywords, not only keywords composed of correct words related to advertising information, but also words that are somewhat similar in consideration of misrecognition of the keywords and voice recognition errors are registered. It is preferable.

  Subsequently, in the PC device 20, the presentation area setting unit 22 generates presentation area data indicating a presentation area that is an outer peripheral area among areas surrounded by each user in accordance with a setting command input from an input device (not shown). Set (ST2). For example, as shown in FIG. 5, the presentation area data indicates a presentation area A that is an outer peripheral side area of the information presentation surface of the information presentation unit 30, and has a long side length Lx, a short side length Ly, and a width. d. Note that the presentation area data may be set to a default value based on past data or the like. In any case, the set presentation area data is sent to the presentation start position calculation unit 23 and the presentation image calculation unit 26.

  In the presentation start position calculation unit 23, the foreground region setting unit 23a sets a plurality of foreground region data indicating the foreground region of each user in the presentation region A indicated by the presenting region data, and the foreground region data for each user Is written in the area information storage unit 23b. For example, as shown in FIG. 6, the front area data includes a long side length s of the front areas UA1,... UA4, and a virtual reference line VL parallel to the long side of the information presentation unit 30 through the information presentation unit center 30o. Angles α1 to α4 between the information presentation unit center 30o and a virtual line connecting the users U1 to U4.

  Here, if there is no utterance of each user U1-U4, the PC apparatus 20 waits until there is an utterance (ST3; No). On the other hand, in the PC device 20, if any user (eg, user U4) has an utterance (ST3; Yes), the voice processing unit 24a has the utterance information corresponding to the voice signal input from each microphone 11-13. Is transmitted to the keyword comparison unit 24b, and the positions of the microphones 11 to 13 and the arrival time of the utterance are transmitted to the presentation start position calculation unit 23.

  In the presentation start position calculation unit 23, the foreground region selection unit 23 c is based on the positions of the microphones 11 to 13 and the arrival times of the utterances to the microphones 11 to 13. From the area data, the area data before the user U4 who made the utterance (hereinafter also referred to as the speaker) U4 is selected.

  Here, a method for selecting the near area data will be described. The voice of the user U4 who has spoken is picked up by a plurality of microphones 11 to 13, and the direction of the speech is estimated from, for example, the volume of the voice and the arrival time from the difference in the voice signal input from the microphones 11 to 13 to the PC device 20. There is a way. The estimated direction has low accuracy, and there is a possibility that the value has a certain width as shown by the broken-line triangle in FIG.

Then, the direction in which the utterance is made and the angles α1 to α4 of the user's U1 to U4 preset in advance are compared, and based on the angle closest to the direction in which the utterance is made, the near area UA4 of the user U4 who has spoken can be selected. . For example, as shown in FIG. 8, when the uttered direction is detected from the virtual reference line VL as the upper right direction β in the drawing, the near area data indicating the near area UA4 corresponding to the angle α4 closest to the upper right direction β is selected. . As shown in FIG. 9, the direction θ based on the positions of the microphones 11 to 13 and the arrival times at the microphones 11 to 13 is, for example, two along the virtual reference line VL as shown in FIG. Can be calculated as θ = cos ⁻¹ (μc / L) from the interval L obtained from the positions of the microphones 11 and 12, the arrival time difference μ between the microphones 11 and 12, and the sound velocity c. Further, the arrival time to the remaining one microphone 13 is used to identify whether the calculated angle θ corresponds to an upper right direction or a lower right direction in FIG. When the arrival time to the remaining one microphone 13 is shorter than the arrival times to the other microphones 11 and 12, the angle θ corresponds to the upper right direction in FIG. When the arrival time to the remaining one microphone 13 is longer than the arrival times to the other microphones 11 and 12, the angle θ corresponds to the lower right direction in FIG.

In any case, the near area selection unit 23c calculates the presentation start position from the selected near area data and sends it to the presentation image calculation unit 26 (ST4).
The keyword comparison unit 24b compares all the search keywords in the advertisement information database 21 with the utterance information received from the voice processing unit 24a. If there is no matching search keyword (ST5), the process ends.

Further, if there is a search keyword that matches the utterance information as a result of the comparison (ST5; Yes), the keyword comparison unit 24b extracts the search keyword from the information database and sends it to the information extraction unit 25.
The information extraction unit 25 extracts advertisement information related to the transmitted search keyword from the advertisement information database 21 (ST6), and transmits this advertisement information to the presentation image calculation unit 26.

  In the presentation image calculation unit 26, the presentation position calculation unit 26a presents the advertisement information so that it moves from the presentation start position based on the presentation area data, the current time, and the presentation start position and circulates over the presentation area. The position is calculated according to the time change (ST7), and the calculated presentation position is sent to the presentation image creation unit 26b.

Based on the presentation position and the advertisement information extracted by the information extraction unit 25, the presentation image creation unit 26b creates presentation image data so as to present the advertisement information at the presentation position (ST8).
The presentation image creation unit 26 b sends the presentation image data to the information presentation unit 30. The presented image data is not limited to including one piece of advertisement information, but may include a plurality of pieces of advertisement information depending on the utterances of the users U1 to U4.

The information presentation unit 30 presents the presentation image data received from the presentation image creation unit 26b (ST9). The above operations of steps ST3 to ST9 are repeated until the end of the presentation operation (for example, the end of the conference) (ST10).
[Example of advertisement information presentation]
When the presentation image data starts to be presented, the advertisement information is presented in the front presentation area A corresponding to the presentation start position while being stopped for a certain period of time.

  After the stop presentation, the advertisement information is presented so as to travel around the rectangular frame-shaped presentation area A based on the presentation position calculated according to the time. This moving speed and stop time may be set in advance. When a plurality of advertisement information is being presented, a presentation position is calculated for each advertisement information.

A specific example of the calculation of the presentation position will be described with reference to FIGS.
When advertisement information Adv is newly extracted from the utterance of a certain user U1, the advertisement information Adv is indicated by a position P0 (x0, y0) on the xy coordinates with the vertical direction as the y-axis and the horizontal direction as the x-axis. Thus, it is presented in the center of the area UA1 before the speaker (time t0). At this time, the advertisement information Adv may be presented at the center of the front area UA1 from the beginning, and is also presented at the center of the front area UA1 by moving slightly from the right end of the front area UA1 to the left side and stopping at the center position. May be. In any case, the advertisement information Adv is presented at the presentation start position P0 (x0, y0) in the center of the near area.

  The advertisement information Adv is stopped and presented at the presentation start position P0 (x0, y0) (time t0 to t1). Thereafter, the advertisement information Adv is presented to move to the left side of the drawing at a constant speed v until it reaches the first corner C2 (L_x, y0) (time t1 to t2). At this time, the presentation position P (x1, y0) is calculated as P (x0 + v × (t−t1), y0) according to a change in time t. That is, the position on the x coordinate changes as x1 = x0 + v × (t−t1). The time t2 is t2 = {(L_x−x0) / v} + t1.

When the advertisement information Adv arrives at the first corner C2 (L_x, y0), the advertisement information Adv is presented by rotating 90 degrees to the right so that the lower side always faces the outside of the presentation area A (time t2). This rotation is performed in the same manner at the other corners C3, C4 and C1.
Next, the advertisement information Adv is presented so as to move at a constant speed v from the corner C2 (L_x, y0) until it arrives at C3 (L_x, L_y) (time t2 to t3). At this time, the presentation position P (L_x, y2) is calculated as P (L_x, y0 + v × (t−t2)) according to a change in time t. That is, the position on the y coordinate changes as y2 = y0 + v × (t−t2). The time t3 is t3 = (L_y / v) + t2.

  After that, similarly, move from corner C3 to C4 (time t3 to t4), rotate 90 degrees to the right and move from corner C4 to C1 (time t4 to t5), rotate 90 degrees to the right and present the original from corner C1. It returns to the start position P (x0, y0) (time t5 to t6) and is erased (time t6). In addition, when continuing presentation after returning to the original presentation start position P (x0, y0), the movement is repeated in the same manner to the corner C2. Further, the advertisement information Adv may return to the original presentation start position, and may end the presentation after being displayed for a certain period of time. Further, during the movement of each advertisement information Adv, the presentation position may be adjusted so that the new advertisement information Adv does not overlap the old advertisement information Adv.

  As described above, according to the present embodiment, among the presentation areas that are outer peripheral areas in the area surrounded by each user, the presentation area is circulated from the presentation start position in the area in front of the user who spoke. The presentation position is calculated according to the time change, presentation image data for presenting the utterance related information related to the user's utterance at the presentation position is created, and the presentation image data is presented.

  At this time, since the utterance related information is presented so as to go around the presentation area, it is possible to cope with the case where each person wants to see the information individually, and the outer peripheral area of the area surrounded by each person is the presentation area. Since there is a presentation start position in the area in front of the user who made the utterance, information can be efficiently presented to the utterer.

  Supplementally, information such as advertisements (banner advertisements) searched based on keywords such as the utterances of each person during the conference will be displayed on the inside of the table surface surrounded by the conference participants (inside the table edge, inside each conference) It is possible to share information such as advertisements among the conference participants including the speaker.

  Further, this embodiment is not a technique different from the conventional technique in which the advertisement information Adv is presented in one place such as one side of the conference room or the center of the table and shared by the users U1 to U4 at the same time. This embodiment is a method of presenting and flowing advertisement information Adv in front of each user U1 to U4 as if it is a rotating sushi, so that the advertisement information Adv is transmitted to each user U1 to U4 without disturbing the conversation. Can be presented individually and shared.

In addition, since the advertisement information Adv is presented in front of each of the users U1 to U4, each user U1 to U4 can easily recognize and easily transfer the user's attention to the advertisement information Adv according to the topic. Can do.
In addition, by setting the center of the area UA4, UA1 in front of the uttered users U4, U1 as the presentation start position, information such as advertisement can be efficiently presented to a more interested target person. Further, when the advertisement information Adv is stopped for a certain period from the start of presentation, the advertisement information Adv can be presented more efficiently.

(Modification of the first embodiment)
The present embodiment may be modified as in the following modified examples (1-1) to (1-7).
(1-1) The advertisement information Adv may be deformed so that the advertisement information Adv does not come out of the presentation area during rotation at each corner. For example, since the corner may be a curved curve, the advertisement information Adv may be transformed so as not to come out of the curved line. As this type of deformation, for example, as shown in FIG. 13, a rainbow-shaped deformation that curves the rectangular advertisement information Adv in an arch shape can be used. In addition, the calculation formula and program of the rainbow-shaped deformation are described in, for example, Masahito Masatatsu, “Detailed Image Processing Programming”, Softbank Creative Corporation, March 30, 2008, first edition, p230-232, 240-243. Has been. In addition, when the rainbow-shaped deformation described in the literature is used as it is, if the rainbow-shaped deformation is performed after the advertisement information Adv is rotated up and down, the advertising information Adv 'spreads downward and the lower side always faces the outside of the presentation area A. The advertisement information Adv ′ can be presented so as not to leave the corner. Further, the rainbow-shaped deformation is not limited to the case where it is applied to the entire advertisement information Adv, but the advertisement information Adv is continuously deformed by adding a case where it is partially applied such as only the left half or the right half of the advertisement information Adv. It may be used as follows.

  (1-2) The information presentation unit 30 is a projector device instead of the display device, and each user U1 to U4 surrounds the information presentation unit 30 instead of the situation surrounding the table. This table presents advertisement information Adv by projecting presentation image data from the projector device.

  Accordingly, the presentation area setting unit 22 calculates presentation area data indicating the presentation area A on the outer periphery side of the table. As a calculation method, for example, there is a method of detecting a table area from a camera image and calculating from the table area. Specifically, as shown in the upper half of FIG. 14, a place where the table T exists is photographed from above by the camera 41, and image information when there is no table T and when there is no table T is a place image in the presentation area setting unit 22. Save in the storage unit 22a. The table area calculation unit 22b calculates a table area from the difference between the two pieces of image information in the place image storage unit 22a, or from preset color information of the table T, and sends it to the presentation area extraction unit 22c. The presentation area extraction unit 22c converts the coordinates of the table area in the image information into actual coordinates, extracts the presentation area A having a width d from the periphery of the table area of the actual coordinates, and calculates presentation area data.

  (1-3) Depending on the information presentation method, there are cases where information cannot be presented on all tables surrounded by conference participants, and therefore the presentable area may be used as a table area. In this case, the presentation area extraction unit 22c sets the shape and size of the inner area of the table area from the table area surrounded by the meeting participant, so that the participants around the inner side of the table surrounded by the meeting participant. The front is the presentation area. The detection of the presentation area A may be performed sequentially, or may be performed only at the beginning or only during adjustment installation.

  (1-4) The table area calculation unit 22b may be omitted, and the presentation area extraction unit 22c may set the presentation area data from the position of each user. The shape and width of the presentation area are set in advance from the input device (not shown) in the presentation area extraction unit 22c as a circular donut shape as shown in FIG. 15, for example, and the presentation area extraction unit 22c stores the location image. Based on the position of each user U1 to U3 obtained from the image information in the section 22a, the center of each position or the center of gravity position is determined as the table area center, and the radius that does not overlap the area where each user U1 to U3 exists There is also a method of setting a donut-shaped region of an R circle. The radius R is smaller than the distance between the center of the table area and the positions of the users U1, U2, and U3 by a predetermined value (eg, the length between the wrist and the elbow: about 20 to 30 cm). This value is calculated by the presentation area extraction unit 22c. In addition, the actual table is a rectangle, but it has been changed to a region with rounded corners, and as each user approaches one side of the table, some regions in the table are designated as the presentation region A '. There is also a method of setting the shape and size of the inner area of the table area in a shape different from that of the table.

(1-5) A method for setting the near area UA4 of the user U4 who has spoken will be described.
As preprocessing, the front areas UA1 to UA4 of all the users U1 to U4 are set, and the angle α1 to α4 at which each user U1 to U4 is located from the virtual reference line VL is detected.
As shown in FIG. 16, the front area setting unit 23 a is configured to display a presentation area (dotted pattern area) and each user U1 to U4 from a camera image obtained by photographing the situation in which each user U1 to U4 surrounds the table from above. An existing region (substantially elliptical region having a convex portion at the center) is extracted.

  The existence area of each user U1 to U4 can be detected from the difference between the image information obtained by photographing the situation when the user is absent by the camera 41 and the image information obtained by photographing the situation after the user is seated by the camera 41. At this time, it is necessary to exclude the table area from the difference detection target so as not to detect the material on the table. However, since the user's head may overlap the table area, it is necessary to set the excluded area inside the presentation area on the outer periphery side of the table.

  Also, assuming that all users are around the table, only the outside of the presentation area A may be a difference detection target. In addition, users who are far from the table may be excluded from the difference detection targets. Alternatively, an approximate value of the size (eg, shoulder width) of each user U1 to U4 may be set in advance, and a difference detection result (eg, material) that is too smaller than the approximate value may be ignored.

  Furthermore, in order to obtain the center (or center of gravity) in the area of only the head and shoulder without the hand area, the existence area of each user U1 to U4 may be extracted in consideration of the color and shape. By detecting the difference set in this way, the existence areas of the users U1 to U4 are sequentially detected, and it is possible to cope with another user joining from the middle of the conference. However, the existence areas of the users U1 to U4 may be detected sequentially, or may be detected only at the beginning or only during adjustment installation. The center (or centroid) of the user's presence area is obtained from the detected user's existence area, and a circle with a radius r overlapping the presentation area A from the center (or centroid) is drawn. This circle has a radius r so that the distance between the positions of two points overlapping the center line of the presentation area A is the user's shoulder width or a value close to the shoulder width. From the distance s between two points where the solid circle and the broken center line intersect, as shown in FIG. 6, the front areas UA1 to UA4 of the users U1 to U4 are set. The distance s is an example of the lateral width of the near areas UA1 to UA4 and does not need to be a strict value. The lateral width of the front areas UA1 to UA4 may be set so as not to be too wide and adjacent to each other while covering the front of each user U1 to U4 to some extent. In addition, each user's front area UA1-UA4 may be performed sequentially, and may be performed only at the beginning or only at the time of adjustment installation.

  (1-6) An example in which the angles α1 to α4 of the users U1 to U4 from the virtual reference line VL are detected is shown. Angles α1 to α4 formed by a line connecting the center of each of the users U1 to U4 obtained from the difference of the image information of the camera 41 and the center of the area surrounded by the presentation area A and the virtual reference line VL can be obtained. That's fine. Corresponding angles α1 to α4 are stored as attached information in the area in front of each user U1 to U4. In addition, as a method of setting the front areas UA1 to UA4 of the users U1 to U4, after obtaining the angles α1 to α4 at which the users U1 to U4 exist, the centers of the users U1 to U4 as shown in FIG. The distance s between two points where the line that forms the angle β widened forward and backward intersects the line connecting the center of the table and the line outside the presentation area A is determined, and the near area is set from the distance s To do. Further, instead of setting from the positions of the users U1 to U4, the near area may be set from the position coordinates of a fixed table or chair.

  (1-7) When a certain user U4 utters after finishing the preprocessing, the direction θ in which this utterance is detected is detected. As the speaker direction detection, for example, as shown in the lower half of FIG. 14, the face images of the users U1 to U4 are captured by a plurality of cameras 42 and 43 or an omnidirectional camera (not shown), and image information is sent to the participants. Cameras 42 and 43 that have been stored in the image storage unit 23d, and the foreground region selection unit 23c detects the opening / closing of the mouths of the users U1 to U4 from the image information in the participant image storage unit 23d, and outputs the image information. The direction θ may be estimated based on the position.

(Second Embodiment)
FIG. 17 is a schematic diagram showing the appearance of the utterance related information presentation device according to the second embodiment of the present invention, FIG. 18 is a schematic diagram showing the configuration of the device, and FIG. 19 is an advertisement used in the device. It is a schematic diagram showing the configuration of an information database. The same parts as those in the above-mentioned drawings are denoted by the same reference numerals, detailed description thereof is omitted, and different parts are mainly described here. In the following embodiments, the same description is omitted.

That is, this embodiment has a configuration in which an information selection / operation unit 50 as a contact position detection unit is added to the first embodiment from the viewpoint of operating the presented advertisement information.
Here, the information selection / operation unit 50 is a touch panel that is arranged on the information presentation surface of the information presentation unit 30 and detects a contact position as a selection position selected by the user, and the detected contact position is the PC device 20. It has a function to send to.

  Accordingly, the advertisement information database 21 stores detailed information (accompanying information) in association with the advertisement information, as shown in FIG. For example, when the advertisement information represents a banner advertisement in which a headline, a title, or the like is written, the image of the product, specifications, contact information, and the like can be used as appropriate. However, the detailed information is an example of the accompanying information, and arbitrary accompanying information such as related information not showing details may be used instead of the information showing the details.

  The presentation image creation unit 26b extracts detailed information from the advertisement information database 21 based on the advertisement information presented on the contact position detected by the information selection / operation unit 50, and presents this detailed information as well. It has a function to create presentation image data. Note that the presentation image creation unit 26b is not limited to presenting detailed information, but presents presentation image data corresponding to an arbitrary operation such as stop of movement, immediate erasure, and change of presentation time according to the movement pattern of the contact position. You may create it. Further, the moving advertisement information may be presented so as to overtake the stopped advertisement information.

  Moreover, as a change of presentation time, what is necessary is just to make presentation time of the advertisement information selected by contact longer than presentation time of the advertisement information which is not touched, for example. For example, the presentation time of the selected advertisement information may be simply reset by setting to delete the advertisement information after a predetermined time from the start of presentation or selection.

These operations can be set so as to change depending on the time of contact with the advertisement information, the change of the contact location, the contact location, the movement of the hand, and the like. A specific presentation method and operation method can be set in advance, and can be set to be sequentially switched.
Next, an utterance related information presentation method by the utterance related information presentation apparatus configured as described above will be described with reference to FIG.

  First, in the PC device 20, by operating an input device such as a keyboard or a mouse (not shown), advertisement information (utterance related information), a search keyword, and detailed information are associated with each other and registered in the advertisement information database 21 (ST1 ′). ). That is, unlike the above, detailed information is also registered.

Subsequently, in the utterance related information presentation device, as described above, it is assumed that steps ST <b> 2 to ST <b> 10 are executed and advertisement information that circulates the presentation area A is being presented to the information presentation unit 30.
At this time, if the selection / operation of the advertisement information is not detected by the information selection / operation unit 50 (ST8a; No), the presentation image creation unit 26b and the presentation position calculated by the presentation image calculation unit 26 as described above Based on the advertisement information extracted by the information extraction unit 25, presentation image data is created so as to present the advertisement information at the presentation position (ST8).

  On the other hand, when selection / operation of advertisement information is detected by the information selection / operation unit 50 (ST8a; Yes), the presentation image creation unit 26b detects by the information selection / operation unit 50 in addition to the above-described presentation of advertisement information. Based on the advertisement information presented on the touched position, detailed information is extracted from the advertisement information database 21, and presentation image data is created so that this detailed information is also presented (ST8b).

Thereby, when selection / operation of advertisement information is detected, presentation of detailed information related to advertisement information to be selected / operated is executed in addition to the above-described presentation of advertisement information.
As described above, according to the present embodiment, in addition to the effects of the first embodiment, the advertisement information displayed around the information presentation unit 30 is selected and the detailed information is presented to discuss in more detail. can do. In addition, it is possible to cope with a case where each user wants to see detailed information individually, discussion can be promoted in relation to the detailed information, and detailed information can be efficiently presented to the selected user.

(Modification of the second embodiment)
In addition, you may combine this embodiment with the modification of 1st Embodiment. Specifically, instead of the information selection / operation unit 50 including the touch panel described above, as shown in FIG. 21, the image information in the location image storage unit 22a obtained by the camera 41 that captures the location of the table T is displayed. Based on the difference between the image information, the function of detecting the overlapping position (selected position selected by the user) of the user's hand and the advertisement information image, and the movement of the hand from the shape of the user's hand obtained from the image information You may deform | transform into the structure provided with information selection / operation part 50 'which has the function to detect. In this case, the camera 41, the location image storage unit 22a, and the information selection / operation unit 50 ′ constitute position detection means.

  Since the presentation area calculation unit 26 calculates the presentation position of each advertisement information according to the time change with respect to the presentation area A and creates the presentation image data, the presentation area calculation unit 26 grasps the advertisement information for each presentation position. Advertisement information that overlaps with the position of the hand can be selected, and detailed information corresponding to the selected advertisement information can be extracted from the advertisement information database 21 and presented.

(Third embodiment)
FIG. 22 is a schematic diagram showing an appearance of an utterance related information presentation device according to the third embodiment of the present invention, and FIG. 23 is a schematic diagram showing a configuration of the device.
That is, this embodiment has a configuration in which a central area calculation unit 27 is added to the second embodiment from the viewpoint of operating advertisement information to be circulated and presenting it in the central area.

  Here, the central area calculation unit 27 calculates a central area among the areas surrounded by the presentation area A based on the presentation area data set in the presentation area setting unit 22, and the central area is calculated as a result of the calculation. It has a function of sending the central area data shown to the presentation image creation unit 26b. The center area data is, for example, the center position and the radius, but is not limited thereto, and may be data indicating the center area.

  Accordingly, the presentation image creation unit 26b moves toward the central region based on the central region data received from the central region calculation unit 27 and the contact position detected by the information selection / operation unit 50. At this time, it has a function of creating presentation image data based on the central area data so that the advertisement information presented on the contact position is presented in the central area.

  In addition, the presentation image creation unit 26b creates presentation image data in which the size and orientation of the advertisement information being presented in the central area is changed according to the detection pattern of the contact position, not limited to the presentation in the central area. Also good. Furthermore, the presentation image creation unit 26b may create the presentation image data so as to present the advertisement information being presented in the central area in the original presentation area A on the basis of the contact position detection pattern. . The detection pattern of the contact position can be set according to the time of contact with the advertisement information, the change of the contact location or the contact location, the movement of the hand, or the like. A specific presentation method and operation method can be set in advance, and can be set to be sequentially switched. For example, by moving a finger touched in order to select certain advertisement information in the direction of the center area, the advertisement information is presented in the center area, and when returning, it moves in the same reverse direction. Alternatively, the advertisement information presented in the central area may be set to be deleted after a certain time from the start of presentation in the central area or after a certain time from the user's contact. The operation of changing the size and orientation of the advertisement information is also performed by the movement of the user's hand. For example, the detection size can be changed by changing the size when touching for a certain period of time, or changing the direction when moving in an arc. Is possible.

Next, the speech related information presentation method by the speech related information presentation device configured as described above will be described with reference to FIG.
First, it is assumed that the PC device 20 executes steps ST1 and ST2 as described above. Further, in the PC device 20, the central area calculation unit 27 calculates the central area of the areas surrounded by the presentation area A based on the presentation area data set in the presentation area setting unit 22, and this calculation As a result, the central area data indicating the central area is sent to the presentation image creation unit 26b.

Subsequently, it is assumed that the utterance related information presentation device performs the above-described steps ST3 to ST8a and ST8 to ST10 and presents advertisement information that circulates in the presentation area A to the information presentation unit 30.
At this time, if the selection / operation of advertisement information is not detected by the information selection / operation unit 50 (ST8a; No), the presentation image creation unit 26b executes step ST8 as described above.

  On the other hand, when the selection / operation of the advertisement information is detected by the information selection / operation unit 50 (ST8a; Yes), the presentation image creation unit 26b displays the advertisement information in the above-described outer-side presentation area in the central area. Switch to presenting advertising information at. That is, when the contact position moves toward the central region based on the central region data received from the central region calculation unit 27 and the contact position detected by the information selection / operation unit 50, the presentation image creation unit 26b Presentation image data is created based on the central area data so that the advertisement information presented on the contact position is presented in the central area (ST8b ′).

  Thereby, when selection / operation of advertisement information is detected, it is possible to stop presenting advertisement information in the outer peripheral side presentation area and present advertisement information to be selected / operated in the central area. . Furthermore, operations such as changing the size and orientation of the advertisement information being presented in the central area can be executed.

Specifically, central area data indicating a central presentation area of a table surrounded by conference participants is calculated in advance. Then, the conference participant selects the advertisement information being presented in the presentation area A, and performs an operation of switching the display to the center.
As described above, according to the present embodiment, in addition to the effects of the second embodiment, the selected advertisement information can be presented in the central area, so that the selected advertisement information can be discussed among the users U1 to U4. it can. In addition, it is possible to cope with the case where each user wants to see the information at the same time, the discussion can be promoted in relation to the information in the central area, and the information can be efficiently presented to each user.

(Modification of the third embodiment)
In addition, you may combine this embodiment with the modification of 2nd Embodiment. Specifically, as described above, instead of the information selection / operation unit 50 formed of a touch panel, the configuration including the camera 41, the location image storage unit 22a, and the information selection / operation unit 50 ′ as illustrated in FIG. Advertisement information may be selected and presented in the central area.

  Note that the method described in the above embodiment includes a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), a magneto-optical disk (MO) as programs that can be executed by a computer. ), And can be distributed in a storage medium such as a semiconductor memory.

  Further, the present invention is not limited to the above-described embodiments and modifications as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments and modifications. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

It is a schematic diagram which shows the external appearance of the speech relevant information presentation apparatus which concerns on the 1st Embodiment of this invention. It is a schematic diagram which shows the structure of the device in the embodiment. It is a schematic diagram which shows the structure of the advertisement information database in the embodiment. It is a flowchart for demonstrating the speech related information presentation method in the embodiment. It is a mimetic diagram for explaining presentation field data in the embodiment. It is a schematic diagram for demonstrating near field data in the embodiment. It is a schematic diagram for demonstrating the selection method of near field data in the embodiment. It is a schematic diagram for demonstrating the selection method of near field data in the embodiment. It is a schematic diagram for demonstrating the selection method of near field data in the embodiment. It is a schematic diagram for demonstrating the selection method of near field data in the embodiment. It is a schematic diagram for demonstrating the specific example of calculation of the presentation position in the embodiment. It is a schematic diagram for demonstrating the specific example of calculation of the presentation position in the embodiment. It is a mimetic diagram showing modification processing of advertising information in a modification of the embodiment. It is a schematic diagram which shows the system using the camera in the modification of the embodiment. It is a mimetic diagram showing a donut type presentation field in a modification of the embodiment. It is a schematic diagram for demonstrating the setting method of the near region in the modification of the embodiment. It is a schematic diagram which shows the external appearance of the speech relevant information presentation apparatus which concerns on the 2nd Embodiment of this invention. It is a schematic diagram which shows the structure of the device in the embodiment. It is a schematic diagram which shows the structure of the advertisement information database in the embodiment. It is a flowchart for demonstrating the speech related information presentation method in the embodiment. It is a schematic diagram which shows the system using the camera in the modification of the embodiment. It is a schematic diagram which shows the external appearance of the speech relevant information presentation apparatus which concerns on the 3rd Embodiment of this invention. It is a schematic diagram which shows the structure of the device in the embodiment. It is a flowchart for demonstrating the speech related information presentation method in the embodiment.

Explanation of symbols

  11 to 13 ... microphone, 20 ... PC device, 21 ... advertisement information database, 22 and 22 '... presentation area setting section, 22a ... place image storage section, 22b ... table area calculation section, 22c ... presentation area extraction section, 23, 23 '... presentation start position calculation unit, 23a ... near region setting unit, 23b ... region information storage unit, 23c ... near region selection unit, 23d ... participant image storage unit, 24 ... search keyword extraction unit, 24a ... sound processing unit 24b ... keyword comparison unit 25 ... information extraction unit 26 ... presentation image calculation unit 26a ... presentation position calculation unit 26b ... presentation image creation unit 27 ... center region calculation unit 30 ... information presentation unit 41-43 ... Camera, 50, 50 '... Information selection / operation unit, U1-U4 ... User, A, A' ... Presentation area, UA1-UA4 ... Front area, VL ... Virtual reference line, α1-α4 ... Angle, C1-C4 ... Corner, T ... Table, Adv, Adv '... Advertising information.

Claims (6)

An utterance related information presentation device for presenting utterance related information related to the utterance of a user who made an utterance among a plurality of users,
An information database in which the utterance related information and the search keyword are stored in association with each other;
A plurality of microphones arranged at different positions, for converting the utterances into audio signals and inputting them,
A presentation area setting means for setting presentation area data indicating a presentation area that is an outer peripheral area of the area surrounded by each user;
Of the presentation areas, a near area setting means for setting a plurality of near area data indicating the near area of each user,
Based on the position of each microphone and the arrival time when the utterance reaches each microphone, the user selects the near area data of the user who made the utterance from the each near area data, and starts presenting from the near area data A presentation start position calculating means for calculating a position;
Search keyword extraction means for comparing all search keywords in the information database with utterance information corresponding to the voice signal, and extracting a search keyword that matches the utterance information from the information database;
Information extracting means for extracting utterance related information related to the extracted search keyword from the information database;
Based on the set presentation area data and the calculated presentation start position, the presentation position of the utterance related information is changed according to time change so as to move from the presentation start position and circulate on the presentation area. Presenting position calculating means for calculating;
Based on the calculated presentation position and the extracted utterance related information, presentation image creation means for creating presentation image data so as to present the utterance related information to the presentation position;
Information presenting means for presenting the presented image data;
An utterance related information presentation device characterized by comprising: In the utterance related information presentation device according to claim 1,
A position detecting unit arranged on the information presenting surface of the information presenting unit and detecting a selected position selected by the user;
The information database stores associated information in association with the utterance related information,
The presentation image creation means extracts accompanying information from the information database based on the utterance related information presented on the detected selected position, and creates presentation image data so that the accompanying information is also presented. An utterance-related information presentation device characterized by: In the utterance related information presentation device according to claim 2,
Based on the presentation area data, a central area calculation means for calculating a central area among the areas surrounded by the presentation area and sending the central area data indicating the central area to the presentation image creation means as the calculation result Is further provided,
When the selected position moves toward the central area based on the central area data and the selected position, the presentation image creating means presents the utterance related information presented on the selected position to the central area. As described above, an utterance related information presentation device that creates presentation image data based on the central area data. An information database in which utterance related information and search keywords are stored in association with each other, a plurality of microphones arranged at different positions, for converting a speech into an audio signal and inputting, a presentation area setting means, and a front area setting Means, a presentation start position calculation means, a search keyword extraction means, an information extraction means, a presentation position calculation means, a presentation image creation means, and an information presentation means, and uttered out of a plurality of users An utterance related information presentation method executed by an utterance related information presentation device for presenting utterance related information related to a user's utterance,
The step of setting the presentation area data indicating a presentation area that is an outer peripheral side area among the areas surrounded by each user;
The step of setting the foreground region setting means to set a plurality of foreground region data indicating the near region of the user among the presentation regions;
The presentation start position calculation means selects the near area data of the user who made the utterance from the near area data based on the position of each microphone and the arrival time when the utterance reached each microphone. , Calculating the presentation start position from the near area data,
The search keyword extraction means compares all search keywords in the information database with utterance information corresponding to the voice signal, and extracts a search keyword that matches the utterance information from the information database;
The information extracting means extracting utterance related information related to the extracted search keyword from the information database;
Based on the set presentation area data and the calculated presentation start position, the presentation position calculation means presents the utterance related information so as to move from the presentation start position and circulate on the presentation area. Calculating the position according to the time change;
The presentation image creating means creating presentation image data based on the calculated presentation position and the extracted utterance related information, so as to present the utterance related information at the presentation position;
The information presenting means presenting the presentation image data;
An utterance related information presentation method characterized by comprising: In the speech related information presentation method according to claim 4,
The utterance related information presentation device further includes a position detection unit that is arranged on an information presentation surface of the information presentation unit and detects a selection position selected by a user.
When accompanying information is stored in the information database in association with the utterance related information,
When the utterance related information is presented at the detected selected position, the presentation image creating means extracts the accompanying information from the information database based on the utterance related information, and also presents the accompanying information. A speech related information presentation method, further comprising a step of creating presentation image data. In the speech related information presentation method according to claim 5,
Based on the presentation area data, the utterance-related information presentation device calculates a central area among the areas surrounded by the presentation area, and creates the presentation image as a result of calculating the central area data indicating the central area. When further comprising a central area calculation means for sending to the means,
When the selected position moves toward the central area based on the central area data and the selected position, the presentation image creating means presents the utterance related information presented at the selected position to the central area. The speech related information presentation method further comprising the step of creating presentation image data based on the central area data.

Images