JP7090116B2 - Program, recording medium, augmented reality presentation device and augmented reality presentation method - Google Patents

Description

The present invention relates to a program, a recording medium, an augmented reality presentation device, and an augmented reality presentation method, and more particularly to a technique for presenting augmented reality via a display unit of a terminal carried by a user.

There is a technique that uses wearable devices to present augmented reality.

In order to avoid complicated user operations, when a wearable device approaches, playback of virtual content that is superimposed on an object in real space and is presented at the corresponding position is started (). Patent Document 1).

Japanese Unexamined Patent Publication No. 2015-307242

In the technique described in Patent Document 1, although the playback control of the content is performed according to the position of the wearable device, the control such as estimating the state of the user and differentiating the playback content of the content is not performed. Further, in Patent Document 1, when an advertisement applied to the exterior of a bus or a train is targeted, the corresponding content is reproduced when the user is close to the user, but when the user is far away from the user, etc. No disclosure was made as to how to control the presentation.

At least one embodiment of the present invention has been made in view of the above-mentioned problems, and is a program or recording medium that estimates the state of the user to be viewed and presents augmented reality in a suitable manner. , An augmented reality presentation device and an augmented reality presentation method.

In order to achieve the above-mentioned object, the program according to at least one embodiment of the present invention has an imaging means, and the captured image obtained by imaging the real space by the imaging means is said to have the same. By superimposing and displaying a character image depicting a virtual character placed in a virtual space associated with the real space, a computer that presents a sense of extended reality can acquire a captured image and a computer in the real space. The process of determining the position and orientation of the viewpoint in the virtual space where the virtual character is drawn based on the position and attitude of the viewpoint, and the process of controlling the action to be performed by the virtual character based on the position and attitude of the viewpoint, and the action The process of drawing the reflected virtual character for the viewpoint to generate a character image, the process of superimposing the character image on the captured image and displaying the generated superimposed image on the display means, and the process of reflecting the action on the virtual character. As a result, it is a program that executes a process of estimating the state of the user who uses the computer based on the virtual character and the viewpoint on which the action is reflected, and is estimated in the result of reflecting the action on the virtual character. The action to be taken by the virtual character is controlled according to the state of the user.

With such a configuration, according to at least one embodiment of the present invention, it is possible to estimate the state of the user to be viewed and to present the augmented reality feeling in a suitable manner.

A block diagram showing a functional configuration of an AR presentation terminal 100 according to an embodiment of the present invention. The figure for demonstrating the real space and the virtual space which provide the viewing experience of the AR content which concerns on embodiment of this invention. The figure which exemplifies the screen which performs augmented reality presentation in AR presentation terminal 100 which concerns on embodiment of this invention. A flowchart illustrating the presentation process executed in the AR presentation terminal 100 according to the embodiment of the present invention. The figure which exemplifies the data structure of the action information managed by the action list in the presentation process which concerns on embodiment of this invention. The figure for demonstrating the screen which performs augmented reality presentation in AR presentation terminal 100 which concerns on modification 2 of this invention.

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In one embodiment described below, an augmented reality (AR:) is achieved by superimposing a computer graphics (CG) image on a live image obtained by imaging as an example of an augmented reality presenting device. Augmented Reality) An example in which the present invention is applied to an AR presentation terminal 100 capable of presenting will be described. However, the present invention is applicable to any device capable of presenting at least a visual augmented reality by superimposing a predetermined image on a live-action image. Further, in the present specification, the "real space" refers to a real three-dimensional space that the user can recognize without using the AR presentation terminal 100, and the "virtual space" refers to the CG constructed in the AR presentation terminal 100. Refers to a three-dimensional space for drawing, and "enhanced real space" is a real space and a virtual space expressed by superimposing an image obtained by drawing a virtual space on a real image obtained by imaging the real space. It will be explained as referring to the space expressed by synthesizing.

<< Functional configuration of AR presentation terminal >>
FIG. 1 is a block diagram showing a functional configuration of an AR presentation terminal 100 according to an embodiment of the present invention.

The control unit 101 is, for example, a CPU and controls the operation of each block of the AR presentation terminal 100. The control unit 101 controls the operation of each block by reading the operation program of each block stored in the recording medium 102 and the program related to the AR presentation application, expanding the program in the memory 103, and executing the program.

The recording medium 102 is a non-volatile recording device that may include, for example, a rewritable built-in memory of the AR presentation terminal 100 or an optical disk that can be read via an HDD or an optical drive. The recording medium 102 records not only the operation program of each block and the program related to the AR presentation application, but also information such as various parameters required for the operation of each block. It is assumed that various data used for the operation of the AR presentation application executed by the AR presentation terminal 100 of the present embodiment are also stored in the recording medium 102. The memory 103 is, for example, a volatile memory, and is not only as an expansion area of an operation program of each block or a program of an AR presentation application, but also as a storage area for temporarily storing intermediate data and the like output in the operation of each block. Is also used.

The image pickup unit 104 is an image pickup device unit having an image pickup element such as a CCD or a CMOS sensor, and functions as an outside world recognition means of the AR presentation terminal 100, not limited to the acquisition of a live-action image used for AR presentation. The image pickup unit 104 takes an image of a subject existing in the real world (real space) and outputs a captured image (actual image). Imaging is performed intermittently, and while the AR presentation application is being executed, the live-action images are sequentially displayed on the display unit 120, which will be described later. Real space + virtual space) can be viewed.

The detection unit 105 applies predetermined image processing to the live-action image output by the image pickup unit 104, and at which position in the real space the AR presentation terminal 100 is present and in what posture it is. Is detected. Prior to providing a viewing experience using the AR presentation application of this embodiment, feature information in the real space to be provided is collected, and calibration is performed to associate the virtual space with the real space. And. This makes it possible to detect the position and orientation of the AR presentation terminal 100 based on the feature information contained in the live-action image. Further, the detection unit 105 does not need to apply image processing to all the live images captured in continuous frames to perform detection, but performs detection to the live images captured at predetermined time intervals, for example, a gyro. This may be supplemented based on the sensor output of the sensor 110 including the sensor, the acceleration sensor and the like. Alternatively, the detection unit 105 may perform detection only by the sensor output of the sensor 110 without using the image processing of the live-action image.

The behavior control unit 106 controls the behavior of the virtual object to be superimposed and presented on the live-action image in the AR presentation application of the present embodiment. The virtual object presented by the AR presentation application is a character (AR character) whose appearance is formed by a three-dimensional model, and the behavior control unit 106 is concerned based on the position and orientation of the AR presentation terminal 100 and other parameters. It controls various actions such as actions and behaviors to be taken by the AR character. In the present embodiment, the action taken by the AR character occurs over a plurality of frames, and not only the action caused by applying the corresponding motion data to the three-dimensional model corresponding to the AR character, but also the action. It shall include the utterance of the dialogue associated with the action or situation. For the sake of simplicity, in the following description, it is assumed that the virtual object superimposed and presented on the live-action image is only an AR character, but the implementation of the present invention is not limited to this.

The presentation control unit 107 controls the presentation of various information to the user in the AR presentation terminal 100. The AR presentation terminal 100 of the present embodiment has a display unit 120 for displaying an image (AR presentation screen, other OS menu screen, etc.) and an audio output unit 130 for audio output as means for presenting various information to the user. However, it goes without saying that the means for presenting information is not limited to these, and alternatives or additions are possible.

The presentation control unit 107 includes a drawing device such as a GPU, and performs a predetermined drawing process when generating an AR presentation screen to be displayed on the display unit 120. Specifically, the presentation control unit 107 creates a three-dimensional model of the AR character based on the processes and commands performed by the control unit 101 and the actions determined by the action control unit 106 during the execution of the AR presentation application. On the other hand, an appropriate arithmetic process is executed, and first, an image related to the virtual space (an image in which only the AR character is presented) is drawn. Then, the presentation control unit 107 generates an AR screen (a screen related to the augmented reality space) that presents an augmented reality feeling by synthesizing the drawn image related to the virtual space and the live-action image related to the real space. The generated AR screen is presented to the user by being output and displayed on the display unit 120 provided in the AR presentation terminal 100. The display unit 120 is a display device included in the AR presentation terminal 100 such as an LCD. In the present embodiment, in consideration of portability at the time of providing the viewing experience, the display unit 120 is described as being built in and integrated with the AR presentation terminal 100, but the implementation of the present invention is not limited to this. However, for example, it may be a display device detachably connected to the outside of the AR presentation terminal 100 regardless of whether it is wired or wireless.

Further, the presentation control unit 107 includes a circuit for outputting / amplifying an audio signal such as a sound board or an amplifier, and performs predetermined processing when generating an audio output from the audio output unit 130. Specifically, the presentation control unit 107 determines the voice data to be output at the same time based on the voice data recorded in advance on the recording medium 102, and converts the voice data into an electrical voice signal (D / A conversion). And output to the audio output unit 130 to output audio. The voice output unit 130 may be a predetermined speaker or the like, and outputs a sound wave based on the input voice signal.

The operation input unit 108 is a user interface included in the AR presentation terminal 100 such as a touch panel or a button. When the operation input unit 108 detects the operation input made by the user, the operation input unit 108 outputs a control signal corresponding to the operation input to the control unit 101.

The communication unit 109 is a communication interface possessed by the AR presentation terminal 100 for communicating with other devices. The communication unit 109 connects to, for example, another server existing on the network by a predetermined communication method regardless of whether it is wired or wireless, and transmits / receives data. Information such as feature information used for the program and detection of the AR presentation application and a scenario describing the basic action transition of the AR character may be configured to be receivable from an external device via the communication unit 109.

<< AR content overview >>
Hereinafter, the outline of the AR content that provides the viewing experience accompanied by the presentation of augmented reality by the AR presentation application executed by the AR presentation terminal 100 of the present embodiment will be described.

<Space setting>
In the present embodiment, the AR content is content in which the AR character guides the user from the storefront to a predetermined position in the store in one store. As shown in FIG. 2A, the range of the real space (the range around the store including the storefront and the inside of the store) that enables the presentation of AR content is associated with the virtual space associated with the range. ..

As shown in FIG. 2A, static (non-moving) objects (real objects) such as walls, signs, stairs, desks, and chairs installed in the real space are superimposed on the live image. In order to preferably realize the shielding expression by the real object of, a three-dimensional object corresponding to the virtual space is arranged. The three-dimensional object is not a drawing target when drawing an AR character that is also arranged in the virtual space, but if it exists closer to the viewpoint from which the drawing is performed than the AR character, the AR character is shielded. It is the target of the depth value comparison for determining the presence or absence of drawing so that the drawn expression is made. Further, these three-dimensional objects in the virtual space are arranged according to the actual size and arrangement relationship of the corresponding real objects, have the same shape as the real objects, and are sized according to a predetermined scale. do.

The virtual space in which the virtual object corresponding to the static real object is placed is configured based on the range that provides the viewing experience in advance, and the calibration for associating the real space with the virtual space is performed prior to the execution of the AR presentation application. It shall be done. That is, before providing the viewing experience using the AR presentation terminal 100, it is defined for drawing according to the arrangement of the real object in the real space with respect to the image pickup unit 104 of the AR presentation terminal 100 and the position and orientation of the AR presentation terminal 100. It is assumed that the translation and rotation of the coordinate system of the virtual space are set so as to match the arrangement of the corresponding virtual object in the associated virtual space with respect to the viewpoint to be performed.

<Presentation of augmented reality>
While the AR presentation application is being executed, the imaging unit 104 intermittently captures images (movie shooting), and the obtained live-action images are sequentially displayed on the display unit 120, so that a so-called through display showing the state of the real space is displayed. It will be realized. Further, when the AR character is included in the image angle corresponding to the imaging range of the live image in the virtual space, the image 300 of the character is superimposed on the live image as shown in FIG. 3 (a). Then, it is possible to present an augmented reality feeling that makes the user perceive as if the AR character exists in the real space. Here, the condition for superimposing the AR character image on the live-action image is that at least a part of the AR character is included in the angle of view of the virtual space corresponding to the imaging range, and the AR character is not necessarily arranged. It is not necessary for the live-action image to include the surface or feature of the real space that serves as a reference for the position.

In order to present the augmented reality of AR content, the viewpoint of drawing a virtual space in synchronization with the movement and posture change of the AR presentation terminal 100 in the real space, more specifically, the movement and posture change of the imaging unit 104. It is necessary to move and change the posture. Therefore, the detection unit 105 detects the position and orientation of the AR presentation terminal 100 based on the live-action images sequentially obtained by imaging and the sensor output of the sensor 110. When the position and posture of the AR presentation terminal 100 in the real space are specified, the position and posture (line-of-sight direction) of the viewpoint for drawing the virtual space are also specified accordingly, so that the virtual space is drawn based on the viewpoint. By superimposing it on the live image, it is possible to generate a screen that presents augmented reality without any discomfort.

In the AR content of this embodiment, since the scenario is based on the theme of "customer service" in which the AR character walks with the user and guides the user into the store, the viewpoint in the virtual space is determined by the user's head. It is assumed that it functions in the same way as an object recognized as a part (face or eyes + line-of-sight direction). That is, the AR character takes an action such as talking to the user's head.

Further, in providing a series of viewing experiences related to AR contents, a route 201 in which the AR character basically travels is preset in the virtual space, as shown in FIG. 2 (b). As described above, the AR content of the present embodiment is content in which the AR character guides (guidances) a user existing in the store (a user using the AR presentation terminal 100) to a predetermined position (target) in the store. A route 201 connecting the area 202a, which is the start position of the guidance, to the area 202d, which is the target position, is set. Further, as shown in the figure, the route 201 may be provided with other points (areas 202b and c) in addition to the points (areas 202a and d) corresponding to the start position and the target position, and each of them may have an AR character. Is associated with an event that causes a predetermined action. In the present embodiment, when the AR presentation terminal 100 enters (or approaches) the real space area corresponding to each area 202, an event that causes the character to take an action occurs.

In order to present a suitable augmented reality, each area 202 is controlled not to be displayed in the augmented reality space so that the user cannot visually recognize its existence. Further, in order to show that the AR character behaves naturally, each area 202 is separated by concentric circles as shown in FIG. 2 (c), and the distance between the center of the area and the viewpoint is separated. It is assumed that the mode of gradually controlling the behavior of the AR character according to the situation is adopted.

Assuming that the suitable occurrence position of the event defined for the area 202 is the inner area 203 shown by hatching in the figure, which is the central part of the area, the user (AR presentation terminal 100) enters the area. The behavior of the AR character is controlled so that the AR character is allowed to do so. More specifically, when the AR presentation terminal 100 enters the outer region 204 defined outside the inner region 203, the AR character is made to take an action to guide the AR to further advance to the inner region 203, which is a suitable event. The user can be guided to the outbreak. For example, when it is detected that the AR presentation terminal 100 has entered the outer area 204, the AR character arranged in the center of the area 202 is made to take actions of "stopping the user" and "prompting the user to approach". It is possible to easily satisfy the event generation condition (the entry of the AR presentation terminal 100 into the inner region 203) defined for the area. Therefore, as shown in the figure, the outer region 204 is configured to have a larger radius range than the inner region 203, and when the AR presenting terminal 100 is present in the region, the inner region 204 controls the behavior of the AR character. It is possible to draw attention to the user existing in the vicinity of the region 203 and guide the user to a naturally suitable event occurrence position.

In other words, in the AR presentation application of the present embodiment, multi-step event generation is defined in each area 202 according to the distance from the center so that the user is guided along the route. In the embodiment of the present embodiment, for each area 202, the generation condition of a series of actions (call → guidance → unique event) of the AR character for experiencing the unique event in the inner region 203 is set to AR in the outer region 204. Although it will be described as being satisfied when the presentation terminal 100 enters, the implementation of the present invention is not limited to this. For example, different unrelated events are assigned to each area separated by concentric circles, and one or more event occurrence conditions can be satisfied at the same time depending on how close to which area. At least one of these may be controlled to be generated according to a predetermined priority or the like. In this case, the information of the event for which the occurrence condition is satisfied is sequentially stacked, and when the condition is satisfied, it is presented in the form of the action of the AR character.

Further, in the present embodiment, the area 202 will be described as being formed of a circle (perfect circle), but any shape such as a rectangle or a polygon may be used. In particular, considering the characteristics of the AR content for customer service, the shape of the area 202 may be an elliptical shape or a fan shape extending in the line-of-sight direction of the AR character.

Further, the AR content may provide an appreciation experience accompanied by not only visual but also auditory augmented reality presentation. If the output from the voice output unit 130 is configured to enable some sound image localization such as stereo and surround settings, for example, the voice vocalization event of the AR character is within the range of the user (AR) of the AR character. The presentation terminal 100) may be configured to utter when captured (exist) to draw the user's attention to the voice source. That is, even if the AR character does not exist in the angle of view displayed by the display unit 120 of the AR presentation terminal 100, the user can be made to recognize the existence of the AR character by the voice output. Therefore, the detection unit 105 may be configured so that the corresponding position in the virtual space can be specified from the feature information included in the live-action image even when the real space in which the AR character does not exist is imaged. ..

Further, in the present embodiment, a method of specifying the position and posture of the AR presentation terminal 100 by analyzing the live image captured by the image pickup unit 104 or the like is adopted, but the position and posture of the AR presentation terminal 100 are specified by a predetermined method. It may be performed by an external device configured to detect the AR presentation terminal 100 existing in the real space range, and may be supplied to the AR presentation terminal 100.

<Experience of viewing AR contents>
Next, the viewing experience of the AR content provided by the AR presentation application of the present embodiment will be described in more detail. For the sake of simplicity, the viewing experience provided will be described below, including the actions of the AR character to be taken according to the positional relationship between the user and the AR character, but in reality, the AR presentation terminal 100 and the AR presentation terminal 100 in the augmented reality space will be described. Needless to say, the action control is performed according to the positional relationship with the AR character or the positional relationship between the viewpoint corresponding to the position and posture of the AR presentation terminal 100 in the virtual space and the AR character.

In the AR presentation application of the present embodiment, for example, it is started when a user approaches an AR character in a store, and a predetermined position in the store from the store (a reception desk in which a clerk who acts as a seat guide in the real world exists, or a receptionist). An experience of viewing a scenario in which the AR character guides the user while leading (or accompanying) the user along a predetermined lead wire (route 201 in FIG. 2 (b)) to the vacant seat) is provided. To. Here, the route 201 is provided only as a reference, and may be slightly changed depending on the movement content of the user.

For example, the user waits in the waiting line at the storefront until it is in order, and when a vacant seat is available in the store, the user receives the AR presentation terminal 100 running the AR presentation application from the clerk. After receiving the AR presentation terminal 100, the user can freely move while viewing the augmented reality space via the display unit 120.

In response to the user approaching the area 202a defined as the start position (entering the outer area 204 of the area 202a) in the area 202 on the route 201 shown in FIG. 2 (b), the AR character is subjected to the AR character. It urges the user to face and approach, and on condition that he / she has approached further (entered the inner area 203), he / she starts greeting the store visit and saying a line to guide the user into the store.

As shown in FIG. 3B, for example, the speech by the AR character is configured as a balloon above the head of the AR character 301 in order to prevent the voice from being overlooked and to clarify which AR character is speaking. This is done by simultaneously presenting the character string 303 of the dialogue content to the flat plate object (speech balloon object 302). Further, since the balloon object 302 does not fit within the angle of view depending on the viewing direction, the subtitle 304 having the same content as the character string 303 may be configured to always be included in the screen.

When the guidance to the store is started, the AR character starts to proceed at a predetermined speed along the route 201. Each time the route is in progress, the AR character makes a statement or action that encourages him to follow, as shown in FIG. 3 (c). The user enters the store in a follow-up manner while viewing the display unit 120.

When the AR character reaches the area 202 set on the route, the AR character waits in the vicinity, and the AR character responds to the user entering the outer area 204 or the inner area 203 of the area. Take action related to the event specified for the area.

By the way, although the AR character moves along the route 201, there is a possibility that the user loses sight of the AR character. Therefore, in the present embodiment, the action control unit 106 is in a state where the user "lost the AR character" based on the distance between the AR character and the viewpoint as a result of reflecting the action related to the guidance (guidance action) in the virtual space. It shall be estimated whether or not it is in. Further, the action control unit 106 controls so as to change the action to be taken by the AR character based on the estimation result.

That is, the AR character not only takes a predetermined action for the area on condition that the user enters the area 202, but also dynamically responds to the distance between the AR character and the user after the predetermined action. It is controlled to take various actions. For example, for an event that occurred in area 202a, when the AR character moves along the route 201 toward the next area 202b, if the distance between the AR character and the user exceeds a predetermined threshold, the user AR. The behavior control unit 106 controls the behavior of the AR character, such as presuming that the character is in a lost state, looking back, stopping and calling, and returning to the route 201 to approach the user according to the distance.

<< Presentation processing >>
The AR character presentation process performed by the AR presentation application of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. The process corresponding to the flowchart can be realized by the control unit 101 reading, for example, the corresponding process program stored in the recording medium 102, expanding it into the memory 103, and executing the process. It should be noted that this presentation process will be described as being started when, for example, in the executed AR presentation application, an operation input related to a request for providing an AR content viewing experience is made. Further, this presentation process exemplifies the process performed for one frame related to AR presentation, and is assumed to be repeatedly executed every frame for continuous presentation.

Further, in the present presentation process, when the event generation condition is satisfied, the action control is basically performed so that the AR character takes at least one of the motion and the voice generation action predetermined for the event. It is assumed that the action is presented via the display unit 120 and the voice output unit 130. The information of each event may be held in the recording medium 102 as data for an AR presentation application, for example, and is associated with the event ID that identifies each event, and when the event occurrence condition is satisfied, the AR character is used. It is assumed that the information describing the behavior including the motion and voice utterance applied to is managed.

In S401, the image pickup unit 104 performs image pickup related to this frame under the control of the control unit 101, and outputs a live-action image.

In S402, the detection unit 105 detects the position and orientation of the AR presentation terminal 100 based on the live-action image captured in S401 and the sensor output of the sensor 110 under the control of the control unit 101. The detected position and orientation may be derived, for example, as the position (coordinates) in the world coordinate system of the virtual space and the rotation angle of each of the three axes around the position. The control unit 101 stores the detected information on the position and posture of the AR presentation terminal 100 in the memory 103 as information on the viewpoint (viewpoint information) for drawing the virtual space.

In S403, the control unit 101 determines whether or not the current viewpoint position has entered the event generation area in any of the areas defined on the route. Whether or not the area has been entered is determined by, for example, whether or not the projection point is included in the area defined for the area when the three-dimensional position indicated by the viewpoint information is projected onto the XZ plane (floor surface in the virtual world). It may be judged by. If the control unit 101 determines that the current viewpoint position has entered the event occurrence area in any area, the process is transferred to S404, and if it is determined that the current viewpoint position has not entered, the process is transferred to S405.

In S404, under the control of the control unit 101, the behavior control unit 106 obtains information on the event that satisfies the occurrence condition among the events associated with the approaching area based on the position and posture of the viewpoint. For example, it is added to the action list held in the memory 103. Further, the action control unit 106 deletes from the action list the information of the event that no longer satisfies the occurrence condition among the information of the event already added to the action list. The action list may be a list that stacks information on events that satisfy the occurrence conditions, and the information (behavior information) of one item in the list is configured to have, for example, the data structure shown in FIG. good.

In the example of FIG. 5, the action information managed as one item of the action list is associated with the item ID 501 for identifying the item, the event ID 502 for identifying the event for which the occurrence condition is satisfied, and the state where the occurrence condition is satisfied. The number of corresponding frames 503 indicating the number of continuous frames, the action flag 504 indicating whether or not the AR character is currently taking the corresponding action (logical type. True, the AR character is performing the corresponding action), and the corresponding action. It may include priority 505 for actions. Therefore, among the events for which the occurrence condition is satisfied in this step, for the event already included in the action list, the action control unit 106 is already in the list instead of the process of adding the action information to the action list. It is assumed that the process of increasing the number of corresponding frames 503 of the action information by 1 is performed. The information of the priority order 505 has a predetermined reference value according to the type of event and may be input as an initial value, but as described later, it depends on the situation of the AR presentation terminal 100 or the AR character. Depending on the situation, it may be configured to be dynamically changeable. Basically, in the action list, the priority 505 related to the action currently taken by the AR character is set to the highest value (priority is given to the action).

Although the details will be described later, the action to be taken by the AR character has a predetermined period until all the actions are completed. Therefore, basically, if there is an action currently applied to the AR character, another action is applied to the AR character until the period required for the action is completed in order to avoid unnatural behavior. It is necessary to control so as not to reflect. On the other hand, as described above, the action to be taken by estimating the user's state (the action when it is estimated that the user has lost sight of the AR character after the start of the guidance action) is transmitted to the user as soon as possible. Is preferable. Therefore, in the present embodiment, in order to take such an action at an appropriate timing, the action control unit 106 does not cause a problem even if the action currently being applied exists, for example, even if the action is interrupted. When it progresses, it is forcibly terminated and the AR character is controlled to take an action based on the user's state estimation. Therefore, the action information further includes an action forced end flag 506 indicating that the action currently being applied to the AR character is ended at a predetermined delimiter. The forced termination flag 506 is, for example, logical type information, and the initial value is added as false, and when it is truly changed, when the corresponding action progresses to a predetermined break, the priority at that time is set. Not limited to this, control is performed so that the action is forcibly terminated and another action is taken as an AR character.

In S405, the action control unit 106 determines whether or not the AR character is currently in a state of being guided. The determination in this step may be made based on whether or not the in-action flag 504 included in the action list indicates whether the true action information indicates the event ID 502 corresponding to the guidance action. If the action control unit 106 determines that the AR character is currently in a state of taking a guiding action, the process is transferred to S406, and if it is determined that the AR character is not in a state of taking a guiding action, the process is set to S408. Transfer.

In S406, the action control unit 106 estimates whether or not the user has lost sight of the AR character based on the information on the position of the AR character and the position of the viewpoint after applying the guidance action in the virtual space. In this embodiment, for the sake of simplicity, only one threshold value for the distance between the viewpoint and the AR character in the virtual space is provided, and when the threshold value is exceeded, it is estimated that the user has lost sight of the AR character. Therefore, the action control unit 106 determines whether or not the distance between the viewpoint and the AR character is separated by a predetermined threshold value or more as a result of the guidance action (behavior accompanied by movement along the route) started by the AR character in the processing of the frame so far. Estimate the user's condition based on. The behavior control unit 106 shifts the process to S407 when it is estimated that the user is in the state of losing sight of the AR character, and shifts the process to S408 when it is estimated that the user is not in the state of losing sight.

In S407, under the control of the control unit 101, the action control unit 106 adds the action information related to the event generated in the situation where the user is presumed to have lost sight of the AR character as a result of the guidance action to the action list. do. Further, the action control unit 106 truly sets a distance flag stored in the memory 103, which indicates that the distance between the viewpoint and the AR character is separated by a predetermined threshold value or more due to movement. The distance flag is changed to false when the distance between the viewpoint and the AR character falls below a predetermined threshold value. Further, when the action currently being applied to the AR character exists, the action control unit 106 truly changes the forced end flag 506 of the corresponding action information (action information in which the action flag 504 is true).

Further, in the present embodiment, as a result of the guidance action as described above, in a situation where the viewpoint and the AR character are separated by a threshold value or more, the action control unit 106 provides behavior information that causes the AR character to take an action to bring the AR character closer to the direction of the viewpoint. Although described as being added to the action list, the implementation of the present invention is not limited to this. That is, in the situation, in order to shorten the distance between the viewpoint and the AR character, at least one of the AR presentation terminal 100 (the user who carries the AR presentation terminal 100) and the AR character is caused to take an action to urge this. All you need is. For example, even if the AR character itself does not move, an action such as a call may be taken so that the user can move the AR presentation terminal 100 closer to the AR character.

In S408, under the control of the control unit 101, the action control unit 106 has a three-dimensional AR character within the angle of view of the viewpoint for drawing the virtual space based on the viewpoint information and the arrangement information of the objects arranged in the virtual space. Determine if at least a portion of the object is included. If the control unit 101 determines that at least a part of the three-dimensional object of the AR character is included in the angle of view, the process is transferred to S409, and if it is determined that the angle of view is not included, the process is transferred to S410.

In S409, the action control unit 106 is stored in the memory 103, and is a logical type information (flag in the angle of view) indicating that the three-dimensional object of the AR character is included in the angle of view of the viewpoint in the virtual space. Is set to true.

On the other hand, when it is determined in S408 that the three-dimensional object of the AR character is not included in the angle of view, the action control unit 106 causes an event (user) generated by not capturing the AR character in the angle of view of the viewpoint in S410. The action information related to the action for attracting the attention of the AR character) is added to the action list. Further, the action control unit 106 sets the flag in the angle of view stored in the memory 103 to false. In this embodiment, for the sake of simplicity, it is assumed that the action information is added and the flag in the angle of view is changed in the frame where it is determined that the AR character is not captured in the angle of view, but the corresponding state is a plurality of frames. It may be judged that this has been achieved by continuing.

In S411, the action control unit 106 determines the priority of the action information included in the action list under the control of the control unit 101. The priority may be determined based on each action information included in the action list, the distance flag, and the flag within the angle of view, and depending on the situation, based on the priority 505 set in the previous frames. It shall be done so that the priorities are exchanged.

For example, in order to avoid the unnatural behavior of the AR character, for an event in which the action flag 504 is a true event, that is, an event in which an action corresponding to the AR character is in progress at least in the immediately preceding frame, the action is in the current frame. If the motion or voice defined for is continued, the action control unit 106 sets the priority order 505 of the action information related to the event to be the highest. The process may be performed by updating the priority order 505, for example, with a predetermined highest priority value. On the other hand, when the forced termination flag 506 of the action information related to the ongoing event is truly set, the priority 505 is set to be the highest for the corresponding action until the forced termination frame. When the frame for forced termination is exceeded, the priority 505 is controlled to be lower than that of the action information related to other actions.

If the distance flag is true, it is presumed that the user has lost sight of the AR character. Therefore, if there is an event in which the action is currently in progress, the action control unit 106 performs the action corresponding to the distance. The priority of information 505 is set to the next highest. In this case, the forced end flag 506 of the action information related to the ongoing event is set to true, for example, the corresponding action is forcibly terminated within a few frames, so that the action information corresponding to the distance to the result of the guidance action is obtained. Priority 505 shall be the highest after the forced termination. Further, if there is no event in which an action is currently in progress, the action control unit 106 may set the priority 505 of the action information registered when the distance flag is set to the highest immediately.

Further, since it is not preferable to proceed with the main event without being captured by the angle of view, the behavior control unit 106 captures the AR character within the angle of view of the viewpoint when the flag in the angle of view is false. Similarly, the priority 505 related to the event generated by not doing so is set high according to the presence or absence of the event in which the action is currently in progress. In this embodiment, the action to be taken when the user is presumed to be in a state of losing sight of the AR character, that is, the action to be taken in response to the distance from the result of the guidance action, is captured by the angle of view. It shall be treated separately from the action to be taken when the in-angle flag is false, as it includes the action for the purpose.

Further, when there is an event in which the AR character has already performed the corresponding action, the action control unit 106 sets the priority order 505 of the corresponding action information to the lowest value so that the same event does not occur. Alternatively, a process of deleting the corresponding action information from the action list may be performed.

The priority of basic events is set for, for example, an event in which an action is currently in progress, an event for eliminating the distance between the viewpoint and the AR character, an event for capturing the AR character within the angle of view, and an area. The behavior control unit 106 may set the priority order 505 in the order of the events. At this time, if there are multiple events of the same classification, the corresponding frame number 503 of each action information is referred to, and the event is controlled to start from the event with the longest number of frames satisfying the generation condition. It may be there.

In S412, the action control unit 106 controls the action of the AR character of this frame based on the priority set in S411 under the control of the control unit 101. More specifically, the behavior control unit 106 supplies the posture information, dialogue and voice information of the AR character in this frame to the presentation control unit 107, and causes the presentation control unit 107 to present the information as appropriate. When the presentation control unit 107 presents the frame (screen, voice), the control unit 101 returns the process to S401.

As described above, according to the augmented reality presentation device of the present embodiment, it is possible to estimate the state of the user to be viewed and present the augmented reality in a suitable manner.

[Modification 1]
In the above-described embodiment, it is assumed that the user has lost sight of the AR character on condition that the distance between the viewpoint and the AR character in the virtual space exceeds one predetermined threshold as a result of applying the guidance behavior. .. However, the situation in which the viewpoint and the AR character are separated from each other in the virtual space as a result of having the AR character take the guidance action in this way is not limited to the situation where the user loses sight of the AR character.

In the mode of providing the viewing experience as described above, since the user can freely move by carrying the AR presentation terminal 100, the user does not necessarily follow the AR character, such as observing the appearance of the store or taking a picture. It may not be possible. In addition, the user's situation may vary, such as the user overtaking the AR character, moving in the wrong direction, or having difficulty in proceeding due to some unforeseen circumstances. Therefore, the behavior control unit 106 estimates the user's state by taking into account not only the distance in the virtual space between the AR character and the viewpoint as a result of reflecting the behavior, but also the sensor output of the AR presentation terminal 100 and the shooting direction. It may be controlled so as to change the action to be performed by the AR character based on the estimation result.

For example, when the viewpoint in the virtual space and the AR character are separated from each other beyond a predetermined threshold value, and the posture of the AR presentation terminal 100 can be determined to be substantially stationary while facing a direction different from the traveling direction related to the guidance. The behavior control unit 106 may presume that the user is gazing at some object in the real space. In this case, the action control unit 106 controls the event generation and the action so as to guide the guidance again while causing the AR character to return the route and generating an event such as asking what the user is paying attention to. It may be there.

Further, for example, when the viewpoint exists in the traveling direction of the AR character in the route to be moved by the guidance action, that is, when the user overtakes the AR character and moves, the action control unit 106 provides prompt guidance by the user. It may be presumed to be the desired state. In this case, the action control unit 106 controls the event generation and the action so that the AR character advances the route to the position of the viewpoint and then leads the route at a speed faster than the route movement speed related to the guidance up to that point. do it.

Further, in the above-described embodiment, the predetermined threshold value is described as one, but the embodiment of the present invention is not limited to this, and the result of the movement along the route is between the viewpoint and the AR character. A plurality of threshold values of the distance generated in the above may be provided, and an estimable user state and a corresponding action may be provided step by step.

[Modification 2]
In the above-described embodiment, basically, it is determined whether to approach the preset area of the viewpoint or to generate an event according to the distance between the viewpoint and the AR character, and the action information is added to the action list. Although it has been determined that registration is to be performed, the implementation of the present invention is not limited to this. The conditions for generating an event do not have to be limited to predetermined ones. For example, when an image of a specific object grasped by machine learning is detected in a live image, the geography of the real space to which the virtual space is associated is When the information is acquired, the behavior control unit 106 may control the AR character to generate an event to start a conversation including a topic related to the object or the area.

For example, the specific object installed in the real space may be an event poster, a product poster, a product itself, or the like posted on the wall surface of the store, and when these are detected, the action control unit 106 may set the behavior control unit 106. Action information that causes the AR character to take episodes related to posters, product promotion, guidance talk to purchase, etc. may be added to the action list. At this time, even if the object is captured within the angle of view, the user is not always paying attention to it. , If it is presumed that this is the focus of attention, the action may be initiated. Further, it may be determined from the sensor output of the sensor 110 or the like whether or not the object is focused, the user's interest / theme may be inferred, and the object may be reflected in the subsequent behavior control.

Also, for example, when the store where the viewing experience is provided is near the sea, the topic of the sea is touched, the topic of the weather is shaken when the sky is caught in the angle of view, and the weather when the weather information is received. The conditions for generating events may be adaptively added or deleted, such as touching on the topic of.

[Modification 3]
By the way, in the mode of providing the user with a viewing experience for customer service as described above, the age and height of the user vary. That is, even if the AR character height and the contents of the AR content are configured assuming the use for an adult of the average height, there is a possibility that this cannot be experienced favorably depending on the user. For example, when used by a short stature child, the AR presentation terminal 100 can always be maintained at a height of several tens of centimeters from the ground surface. Therefore, when the AR presentation terminal 100 is kept horizontal, there is a possibility that only the feet of the AR character are presented as shown in FIG. 6A and the AR content cannot be grasped appropriately. Alternatively, when the AR presentation terminal 100 is gripped so as to maintain an elevation angle equal to or higher than the threshold in order to present the face of the AR character, safety is not guaranteed because it is difficult to check the user's own feet, and necessary features. Since it is difficult for information to be included in the angle of view, it may not be possible to stably present AR content.

Therefore, in this modification, the behavior control unit 106 estimates what kind of person the user carrying the AR presentation terminal 100 is, and controls to change the behavior accordingly. More specifically, the behavior control unit 106 estimates the height and age of the user based on the analysis of the live-action image by the detection unit 105 and the sensor output related to the posture of the AR presentation terminal 100, and makes the guidance method by the AR character different. Behavior control is performed.

For example, it is assumed that an AR character whose height is set to 170 cm and whose tone is set to be friendly is used to provide an viewing experience accompanied by an augmented reality presentation. When it can be determined that the elevation angle is equal to or higher than a predetermined degree and the height of the AR presentation terminal 100 from the ground surface is low in the posture changed so that the face of the AR character enters the angle of view in response to the call from the AR character. , The behavior control unit 106 estimates the user as a short stature child. In this case, the behavior control unit 106 may change the behavioral standard of the AR character, such as bending over to talk to the user, setting the tone to a polite language, and slowing down the walking speed, as shown in FIG. 6B. .. Further, even when the product is introduced by recognizing the outside world as in the first modification, the control may be performed so as to shift the target to a product that is easily received by a younger customer base. Similarly, when the height is higher than that of the AR character, that is, the depression angle is shown in the posture changed so that the face of the AR character enters the angle of view, it is determined that the height of the AR presentation terminal 100 from the ground surface is high. If possible, the behavior control unit 106 may perform behavior control such as talking by looking up at the user.

[Other embodiments]
The present invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the present invention. The augmented reality presentation device according to the present invention can also be realized by a program that causes one or more computers to function as the augmented reality presentation device. The program can be provided / distributed by being recorded on a computer-readable recording medium or through a telecommunication line.

100: AR presentation terminal, 101: control unit, 102: recording medium, 103: memory, 104: image pickup unit, 105: detection unit, 106: behavior control unit, 107: presentation control unit, 108: operation input unit, 109: Communication unit, 110: Sensor, 120: Display unit, 130: Audio output unit

Claims (12)

A character image obtained by drawing a virtual character arranged in a virtual space associated with the real space is superimposed on an image obtained by having the image pickup means and the real space being imaged by the image pickup means. By displaying it on a computer that presents augmented reality,
The process of acquiring the captured image and
A process of determining the position and orientation of a viewpoint in the virtual space for drawing the virtual character based on the position and orientation of the computer in the real space.
The process of estimating at least one of the user's height and age,
A process of controlling the action to be taken by the virtual character based on the position and posture of the viewpoint and the estimation result of at least one of the height and age of the user .
A process of drawing the virtual character reflecting the behavior with respect to the viewpoint to generate the character image.
A process of superimposing the character image on the captured image and displaying the generated superimposed image on the display means.
As a result of reflecting the action on the virtual character, a process of estimating the state of the user who uses the computer based on the virtual character on which the action is reflected and the viewpoint.
Is a program that executes
A program that controls an action to be taken by the virtual character according to the state of the user estimated as a result of reflecting the action on the virtual character. As a result of reflecting the action on the virtual character, when the distance between the virtual character on which the action is reflected and the viewpoint exceeds a predetermined threshold, it is estimated that the user's state is in a specific state, and the above is described. The program according to claim 1, wherein the behavior of the virtual character is controlled to be different depending on the distance between the virtual character and the viewpoint. The action to be taken by the virtual character includes an action to move in the virtual space.
The second aspect of the present invention is to make the virtual character perform an action to reduce the distance when the distance between the virtual character and the viewpoint after the movement exceeds the predetermined distance as a result of reflecting the movement on the virtual character. The program described. The program according to claim 3, wherein the action of reducing the distance is at least one of an action of bringing the virtual character closer to the viewpoint in the virtual space and an action of promoting the movement of the computer in the real space. A plurality of the predetermined threshold values are set, and the predetermined threshold values are set.
As a result of reflecting the action on the virtual character in the process of estimating the state of the user, the state of the user changes depending on which threshold value the distance between the virtual character on which the action is reflected and the viewpoint exceeds. The program according to any one of claims 2 to 4. The program according to claim 5, wherein in the process of estimating the state of the user, the posture of the computer is further taken into consideration to estimate the state of the user. The program according to any one of claims 1 to 6, wherein the program further causes the computer to perform a process of detecting the position and posture of the computer in the real space based on the acquired image. .. In the process of estimating at least one of the height and age of the user, the posture of the computer changed to the call to the user by the virtual character is an elevation angle exceeding a predetermined degree, and the position is from the ground surface. The program according to any one of claims 1 to 7, wherein the user is presumed to be a short stature child when the distance is less than a predetermined distance. The program according to claim 8, wherein the behavioral standard of the behavior to be taken by the virtual character is changed when the user is presumed to be a short stature child in the process of controlling the behavior to be taken by the virtual character. .. A computer-readable recording medium on which the program according to any one of claims 1 to 9 is recorded. It has an image pickup means, and a character image depicting a virtual character arranged in a virtual space associated with the real space is superimposed on an image captured image obtained by imaging the real space by the image pickup means. It is an augmented reality presentation device that presents augmented reality by displaying it.
The acquisition means for acquiring the captured image and
A determination means for determining the position and posture of the viewpoint of the virtual space in which the virtual character is drawn based on the position and posture of the augmented reality presenting device in the real space.
A first estimation means for estimating at least one of the user's height and age,
A control means for controlling the action to be performed by the virtual character based on the position and posture of the viewpoint and the estimation result by the first estimation means .
A generation means for generating the character image by drawing the virtual character reflecting the behavior with respect to the viewpoint.
A display control means for displaying the superimposed image generated by superimposing the character image on the captured image on the display means,
As a result of reflecting the action on the virtual character, a second estimation means for estimating the state of the user using the augmented reality presenting device based on the virtual character on which the action is reflected and the viewpoint. Have,
The control means is an augmented reality presentation device that controls an action to be taken by the virtual character according to the state of the user estimated by the second estimation means as a result of reflecting the action on the virtual character. By superimposing and displaying a character image in which a virtual character arranged in the virtual space associated with the real space is drawn on the captured image obtained by capturing the real space by the imaging means. It is an augmented reality presentation method that presents augmented reality.
The acquisition process for acquiring the captured image and
A determination step of determining the position and orientation of the viewpoint of the virtual space in which the virtual character is drawn based on the position and orientation of the terminal having the image pickup means in the real space.
A first estimation step for estimating at least one of the user's height and age,
A control process that controls an action to be performed by the virtual character based on the position and posture of the viewpoint, and
A generation step of drawing the virtual character reflecting the behavior with respect to the viewpoint to generate the character image, and
A display control step of superimposing the character image on the captured image and displaying the generated superimposed image on the display means.
As a result of reflecting the action on the virtual character, it has a second estimation step of estimating the state of the user who uses the terminal based on the virtual character on which the action is reflected and the viewpoint.
An augmented reality presentation method in which, as a result of reflecting an action on the virtual character in the control step, the action to be taken by the virtual character is controlled according to the state of the user estimated in the second estimation step.

Images