JP7630469B2 - VIRTUAL SPACE MOVING DEVICE, VIRTUAL SPACE MOVING METHOD, AND VIRTUAL SPACE MOVING PROGRAM - Google Patents

Description

The present invention relates to a virtual space movement device, a virtual space movement method, and a virtual space movement program.

In recent years, the fat burning effect of aerobic exercise has been said to be very effective not only for dieting but also for improving health. This is because the oxygen absorbed into the blood by aerobic exercise promotes fat burning. In addition, continued exercise increases muscle mass, which in turn increases basal metabolic rate, creating a virtuous cycle that burns more fat. There are various methods of aerobic exercise, such as jogging and swimming, but walking is particularly popular because it is easy to do. However, as the elderly suffer from a decline in their ability to detect danger, including a decline in eyesight, there is an increasing need for technology that can support their movement. From the perspective of efficiently burning fat through aerobic exercise, it is most important to absorb oxygen into the blood, and care must be taken to avoid exercising anaerobicly by going too fast.

Therefore, the first thing that is required is to know the load caused by exercise. Since the load caused by exercise differs from person to person depending on physical strength, etc., it is effective to use the heart rate (pulse rate), which is an individual's biometric information, as a guide, rather than the objective amount of exercise. This is why many athletes train while watching their heart rate. The ideal heart rate for burning fat is generally considered to be around 120, about 70% of the maximum heart rate, although this depends on age, etc. For this reason, various mobile devices that can measure biometric information such as heart rate have been proposed.

For example, Patent Document 1 describes technology relating to an information and communication terminal, such as a mobile phone, that can measure biometric information such as heart rate, and a biometric information system. With this technology, biometric information that cannot be measured by the information and communication terminal can also be stored and managed in the information and communication terminal.

Patent document 2 also describes a portable vital sign monitor, and in particular a technology relating to a portable vital sign monitor that uses a blood flow meter to measure information about the blood flow in the target vital sign by using scattered light from the vital sign. A wristwatch with a heart rate monitor function is also proposed.

For example, Patent Document 3 describes technology related to a wristwatch-type exercise measurement device that can measure exercise intensity and heart rate while worn on the wrist. This technology measures heart rate based on the cardiac potential from a pair of electrodes, and the same principle can also be applied to mobile terminals. A mobility support device equipped with a heart rate monitor has also been proposed.

For example, Patent Document 4 describes a portable navigation system, particularly a technology that provides information that can contribute to health promotion. This technology can associate calorie consumption calculated from heart rate with map data and display it for each road section, thereby supporting outdoor fitness activities in a more enjoyable and effective way. However, this technology only displays "which parts of the map contributed to calorie consumption and which were difficult," and is intended to retroactively evaluate calorie consumption through exercise. Meanwhile, a mobile terminal device that uses augmented reality technology has also been proposed.

For example, Patent Document 5 describes a technology relating to an annotation display system and method that can change the amount of annotation information displayed on a mobile terminal depending on the moving speed of the mobile terminal.

That is, the virtual space movement device according to the first aspect comprises a memory unit that stores movement course information including the scenery when a character moves within the virtual space , an acquisition unit that acquires motion information indicating the amount of motion of a user, a calculation unit that calculates movement quality information that evaluates the user's motion state based on the motion information and the movement state of a character moving within the virtual space in conjunction with the user, and a control unit that controls the movement course information by changing the viewpoint within the virtual space based on the movement quality information.

JP 2014-120006 A Patent No. 4119862 Patent No. 4685705 Patent No. 4822208 JP 2007-292505 A JP 2011-123807 A Patent No. 4767816

However, the conventional technology has the following problems. First, the problems when used as a pacemaker will be described. The method of displaying bioinformation such as heart rate in numbers, as in the technology described in Patent Documents 1 to 3, is not only not suitable for the user to intuitively recognize his/her own pace, but also requires continuous confirmation of the measurement results in order to feed back the measured bioinformation to the user and control the exercise load. This not only increases the time the user spends looking at the display, but also blocks the view from the mobile terminal, which may lead to unexpected accidents due to carelessness of the feet and surroundings. Furthermore, even if the user can check bioinformation such as heart rate, if the user tries to speed up or slow down his/her own pace based on the results, it is difficult for the methods described in Patent Documents 4 to 7 to intuitively and in real time to recognize whether the user's moving speed is fast or slow, or how much faster or slower it is. Furthermore, moving while looking at a mobile terminal is boring and lacks fun.

On the other hand, walking with pets or other people is more enjoyable and easier to continue than walking alone, but in reality, keeping a pet is hard work, and when walking with other people, it can be difficult to maintain your own pace suitable for aerobic exercise if there is a difference in physical strength.

Next, problems when used as a portable navigation system will be described. The method of displaying a map as in Patent Document 4 or Patent Document 6, and the method of displaying an arrow or the like as in Patent Document 5 or Patent Document 7 require an operation to return to the guide route when the direction of travel is lost. In addition, in the case of the method of displaying the user's position on a map, it is difficult for children who are not yet able to read a map and visually impaired people to intuitively recognize the direction of travel. Furthermore, when traveling long distances, in order to reach the destination without giving up on the way, it is essential to have a pace distribution or marching plan, which specifies which sections to travel and how long it will take. However, conventional navigation systems were not able to present the optimal travel pace or planned travel speed in real time. Therefore, the present invention aims to solve the problems of the conventional technology and provide a virtual space travel device that uses augmented reality technology to function as a pacemaker for continuing exercise while having fun under normal circumstances.

That is, the virtual space movement device according to the first aspect includes a storage unit that stores movement course information including the scenery when a character moves in the virtual space, an acquisition unit that acquires bioinformation indicating the biometric condition of an exercising user and exercise amount information indicating the amount of exercise of the user, a calculation unit that calculates movement quality information indicating the quality of the user's exercise and the quality of movement of a character linked to the user based on the bioinformation and the exercise amount information, and a control unit that controls the movement course information by changing the viewpoint in the virtual space based on the movement quality information.

In a second aspect, in the virtual space movement device according to the first aspect, the movement quality information is an evaluation of the user's motion state and the character's motion state, and the control unit may control the movement course information based on the information on the user's motion state and the character's motion state.

A third aspect is a virtual space movement device according to the second aspect, wherein the virtual space movement device is provided with an output unit that outputs movement course information and a character, and the control unit may change the viewpoint and change light source settings of the movement course information output to the output unit.
In a fourth aspect, in the virtual space movement device according to the third aspect, the control unit may change the viewpoint and control a sound effect output to the output unit.

In a fifth aspect, the virtual space movement device may be equipped with shoes that apply an impact to the soles of the user's feet, and the control unit may control the type, magnitude, and timing of the shoe impact based on the movement quality information and the movement course information.

In a sixth aspect, the output unit is an odor generating device that generates multiple types of odors, and the odor generating device may select and generate a specific odor from the multiple types of odors based on the movement quality information.

A seventh aspect is a virtual space movement device according to the third aspect, wherein the virtual space movement device is provided with a purchasing unit that enables the purchase of provable accessory information using a non-fungible token attached to a character, and the memory unit stores at least the movement information linked to the accessory information, and the control unit may control the output unit to attach the stored accessory information to the character while superimposing it on movement course information.

In an eighth aspect, in the virtual space movement device according to the seventh aspect, the purchase unit enables the user to update the smart contract and issues a command to store the updated smart contract in the storage unit, and the output unit may output the smart contract stored in the storage unit.

In a ninth aspect, the virtual space movement device according to the third aspect may further include a direction instruction unit that instructs the user, via the output unit, on the direction of movement of the character based on movement course information about the character's movement.

A tenth aspect is a virtual space movement device according to the seventh aspect, further comprising a connection unit that connects the virtual space with another virtual space, and character or accessory information is output in the other virtual space, and the storage unit may store, as history information, the fact that the character or accessory information has been used in the other virtual space.

The virtual space movement method according to the eleventh aspect includes a storage step in which a computer stores movement course information including the scenery when a character moves in the virtual space , an acquisition step in which motion information indicating a user's amount of motion is acquired, a calculation step in which movement quality information is calculated by evaluating the user's motion state based on the motion information and the movement state of a character moving in the virtual space in conjunction with the user, and a control step in which the movement course information is controlled by changing the viewpoint in the virtual space based on the movement quality information.

The virtual space movement program of the twelfth aspect provides a computer with a memory function for storing movement course information including the scenery when a character moves within the virtual space , an acquisition function for acquiring motion information indicating the amount of motion of the user, a calculation function for calculating movement quality information that evaluates the user's motion state based on the motion information and the movement state of the character moving within the virtual space in conjunction with the user, and a control function for controlling the movement course information by changing the viewpoint within the virtual space based on the movement quality information.

A virtual space movement device etc. according to the present disclosure includes a memory unit that stores movement course information including the scenery when a character moves within the virtual space, an acquisition unit that acquires exercise information indicating the user 's amount of exercise, a calculation unit that calculates movement quality information that evaluates the user's exercise state based on the exercise information and the movement state of the character moving within the virtual space in conjunction with the user, and a control unit that controls the movement course information by changing the viewpoint within the virtual space based on the movement quality information, so that the user can exercise enjoyably and sustainably without getting bored.

FIG. 2 is a block diagram for explaining the functional configuration of the virtual space movement device according to the present embodiment. 4 is a flowchart of a virtual space movement program according to the present embodiment.

The functional configuration of a virtual space movement device 10 according to this embodiment will be described with reference to Fig. 1. The virtual space movement device 10 includes a storage unit 100, an acquisition unit 200, a calculation unit 300, a control unit 400, an output unit 500, a purchase unit 600, a direction indication unit 700, and a connection unit 800.
The storage unit 100 stores movement course information 14 including the scenery when the character 13 moves in the virtual space.
The acquisition unit 200 acquires bio-information indicating the bio-condition of an exercising user and exercise amount information (exercise information) indicating the amount of exercise of the user.

The calculation unit 300 calculates movement quality information indicating the quality of the user's movement and the quality of the movement of the character 13 linked to the user based on the bio-information and the movement amount information (movement information) .
The movement quality information is an evaluation of the user's motion state and the character 13's movement state.
The control unit 400 controls the movement course information 14 by changing the viewpoint in the virtual space based on the movement quality information.
The control unit 400 controls the movement course information 14 based on information on the user's motion state and the character's 13 movement state.
The output unit 500 outputs the movement course information 14 and the character 13 .
The control unit 400 changes the viewpoint and also changes the light source settings of the movement course information 14 output to the output unit 500 .
The control unit 400 changes the viewpoint and controls the sound effects output to the output unit 500 .

The virtual space movement device 10 is equipped with shoes 11 that deliver impacts to the soles of the user's feet, and the control unit 400 controls the type, magnitude, and timing of the impact of the shoes 11 based on movement quality information and movement course information 14.

The output unit 500 is an odor generating device 12 that generates multiple types of odors, and the odor generating device 12 selects and generates a specific odor from the multiple types of odors based on the movement quality information.

The virtual space movement device 10 is equipped with a purchasing unit 600 that makes it possible to purchase provable accessory information using non-fungible tokens attached to the character 13, and the memory unit 100 stores at least momentum information (motion information) by linking it to the accessory information, and the control unit 400 controls the output unit 500 to attach the stored accessory information to the character 13 while superimposing it on the movement course information 14.
The purchasing unit 600 enables the user to update the smart contract he or she has signed up for, and issues a command to store the updated smart contract in the storage unit 100.
The output unit 500 outputs the smart contract stored in the memory unit 100.
The direction indication unit 700 indicates the movement direction to the user via the output unit 500 based on the movement course information of the character's movement.
The connection unit 800 connects a virtual space to another virtual space.

The character 13 or accessory information is output in another virtual space, and the storage unit 100 stores as history information that the character 13 or accessory information 15 has been used in the other virtual space.

Next, the virtual space movement method according to this embodiment will be described together with the virtual space movement program with reference to FIG. 2. FIG. 2 is a flowchart of the virtual space movement program according to this embodiment.

As shown in FIG. 2, the virtual space movement program includes a storage step S100, an acquisition step S200, a calculation step S300, a control step S400, an output step S500, a purchase step S600, a direction indication step S700, and a connection step S800.

The virtual space movement device 10 loads the operating space movement program stored in the ROM or memory unit 100 (not shown) into the main memory, and executes the virtual space movement program using the CPU (not shown).

The virtual space movement program enables the CPU of the virtual space movement device 10 to realize functions such as a memory function, an acquisition function, a calculation function, a control function, an output function, a purchase function, a direction indication function, and a connection function.

These functions are illustrated as being processed in the order shown in the flowchart of Figure 2, but this is not limiting, and the virtual space movement program may be executed by changing the order as appropriate.

Note that the above-mentioned functions overlap with the explanations of the memory unit 100, acquisition unit 200, calculation unit 300, control unit 400, output unit 500, purchase unit 600, direction indication unit 700, and connection unit 800 of the virtual space movement device 10 described above, so detailed explanations thereof will be omitted.
The storage function stores information about a course of movement including the scenery along which the character moves in the virtual space (S100: storage step).
The acquisition function acquires biological information indicating the biological condition of the user who is exercising and exercise amount information indicating the amount of exercise of the user (S200: acquisition step).

The calculation function calculates movement quality information indicating the quality of the user's movement and the quality of movement of a character linked to the user based on the bio-information and movement amount information (movement information) (S300: calculation step).
The control function controls the movement course information by changing the viewpoint in the virtual space based on the movement quality information (S400: control step).
The output function outputs the travel course information and the character (S500: output step).
The purchasing function enables the purchase of provable accessory information by a non-fungible token attached to a character (S500: purchasing step).
The direction instruction function instructs the user on the direction of movement via the output function based on the movement course information of the character (S700: direction instruction step).
The connection function connects a virtual space with another virtual space (S800: connection step).

Below, the virtual space movement device according to this embodiment will be described. In this embodiment, the user selects a walking course in a virtual space. A sensor worn on the body acquires motion information or biometric information when the user actually walks. The motion information includes pace, number of steps, posture, heart rate, ground reaction force, electromyographic signals, joint accuracy, acceleration, and the like. A virtual transformation distance in the virtual environment is calculated from the motion information acquired by the sensor. The virtual transformation distance is calculated and corrected for walking quality based on the biometric information. A virtual camera in the virtual space is moved from a first position to a second position according to the virtual transformation distance. An image acquired by the virtual camera is displayed on a display such as a smartphone.

In this embodiment, the camera and light source in the virtual space move based on the motion information or bio-information. This movement may be controlled to change the sound. The camera angle, light source (lighting), and sound effects are changed based on the user's actions.

In this embodiment, the conditions of the virtual space change depending on the product (shoes, caps, etc.). For example, depending on the shoes, you can run on a new course, in a new season, or at a new time of day, allowing you to experience a different virtual space. By combining shoes and caps, you can run on your favorite course in your favorite season. Select shoes or caps as if you were obtaining an item by paying, and experience walking.

In this embodiment, the route guided in the real world is changed depending on the road conditions in the virtual space. For example, when crossing the mountains of Hakone in the virtual space, the driver is guided (introduced) to a tough route in the real world. The driver cannot clear the route unless he or she passes through the tough route.

In this embodiment, the walking speed is changed according to the road conditions in the virtual space. For example, when walking up a slope in the virtual space, the speed and angle of the treadmill are increased to provide a load equivalent to that of walking up a slope. Rhythmic sounds and weak electric currents are also acceptable. The tempo of the sound and stimulation is increased. The number of steps and thigh-raising movements (that can be cleared) is changed according to the road conditions in the virtual space. A load is imposed on the user in the real world that corresponds to the calories burned on the road conditions in the virtual space.

In light of the fact that working from home has become mainstream, this embodiment senses the real world and changes the virtual space in response to jogging, lifting your legs, moving around, etc. within the home. The movement of going up and down the stairs at home is converted into game movement, actions, etc. within the virtual space. Going up and down the stairs at home becomes more fun.

In this embodiment, you take a walk in a virtual space with a character that resembles a pet (dog, cat). Your pet grows as you take walks (exercise). You can raise a pet in a virtual space. You walk together with a being that is not your reflection (avatar), a supportive object (such as a pet). You walk, stop, take detours, rest, feed, run, etc. for this being. You walk with a being that places restrictions on your actions.

In this embodiment, a virtual space is created that utilizes the human tendency to walk. For example, a golf course, a game of tag, a walk with a child, rescuing a person in distress, or a walk with your child in the virtual space. Walking and running while chasing your child.
In this embodiment, a virtual space is created using images taken by the user. The virtual space is created as the user walks, and as the number of users increases, the virtual space is updated.

In this embodiment, a virtual space is created by stitching together photos of the route the user walked. The photos may be those of a third party (for example, photos from Google (registered trademark) Maps) or may be photos taken by the user. The course the user actually walked is stored in GPS and displayed on Google Maps. The scenery is initially set, but changes to the scenery can be controlled as the user walks. The scenery changes as the number of participants increases.

In this embodiment, rather than creating a virtual space, it is created regardless of one's will. When there are few users in the virtual space, it is unexplored territory, but as the number of users increases, the virtual space is created from the photos brought in by users and selected by users. The scenery and landscape change according to the user's actions and behavior.

In this embodiment, a virtual space is constructed beyond one's control, but one actively constructs the virtual space by cooperating with others of one's own will. Basically, a virtual space is constructed from one's own worldview (photos one has taken, selected), but by collaborating with others, one's worldview (photos one has taken, selected) is combined with and mixed to construct a virtual space. A virtual space is created through collaboration. A walking worldview (virtual space) is created through collaboration. One can bring together photos, videos, and music to create a virtual space to walk in.

In this embodiment, you walk in a virtual space that you have created by yourself or with others (collaboration). You walk as you create it. You take a photo every day and create a virtual space with those photos. You walk in a virtual world based on a diary. By walking in this virtual space, you can reminisce about your past. You can walk while reminiscing about the world of others, your partner, your parents, and your children.

In this embodiment, a walking course in the virtual world is created by walking in the real world. Walking in the real world while recording video, and a walking course in the virtual space is created based on this video.

In this embodiment, we want to link purchased shoes to the quality of walking. A sensor that works with an app is built into the shoes. A pressure sensor is built into the soles of the shoes on a mountain climbing course (virtual space) to measure the load on the soles of the feet when reaching the summit. A sensor that measures the pulling force is built into the shoelaces to measure the force applied when stepping on the foot when reaching the summit. The load on the ankles, knees, etc. is measured.

In this embodiment, the situation in the virtual space changes depending on the type of shoes. If you buy business shoes, you walk in the virtual world of Marunouchi and Kasumigaseki. If you buy golf shoes, you walk on the golf course. If you buy running shoes for the gym, you walk inside the gym facilities.

In this embodiment, shoes linked to virtual space data are sold at an online shop. Purchasers purchase the shoes for the purpose of walking in a new virtual space.

Also, a shared platform for running games using virtual space may be provided separately, and the virtual space movement of this embodiment may be incorporated into the game.

In this embodiment, after hard exercise, the user can select shoes, clothing, etc. For example, after performing high-stress exercise or activities, or running a lot, the user can purchase items such as shoes and clothing in the game. The types of items that can be selected change depending on the amount of exercise strain. If the user is successful in losing weight and reaches their goal, they can make a payment. For example, in a "walking game," the user can improve their walking and running skills by encountering bosses and nationally ranked athletes while walking, allowing the user to progress through the game advantageously and efficiently. The user is not forced to skip walking, and is given the opportunity to enjoy the game advantageously and efficiently. A system in which game settings change according to the amount of strain, and coins are given according to the amount of walking.

For example, measures are needed to anticipate accidents such as bumping into someone while lifting your legs. It would be good to have devices, mechanisms, and methods that can curb and prevent people from becoming too engrossed in the virtual space and causing trouble in the real world. Give users a reason (purpose) to walk. The most common reason is to "maintain health," but an easy way is to change their mood and reset their mind. With remote work becoming more widespread, many users are looking for a change of mood and to reset their mind.

This technology displays AR tags representing disaster information and evacuation routes, and was able to provide appropriate disaster prevention information in real time to users in a small area. However, the evacuation route is displayed as an AR tag of an arrow, and a guide such as an electronic pet or a virtual person is displayed. Furthermore, this guide is displayed by moving away or approaching depending on the user's heart rate and moving speed, so that the user can intuitively recognize the evacuation route and the optimal moving speed. This technology differs from the present embodiment in the features of the technical idea of the invention (problem to be solved, means for solving the problem, and effect). In addition, according to the activity measurement system of the fifth embodiment, the server transmits warning information to the information terminal of a user who may have fallen, and the information terminal that receives the warning information sounds an alarm. Therefore, if the user actually falls and is injured, if the alarm is not turned off, a person near the user (who does not have to be a user of the activity measurement system) can notice and rescue the user. Also, for example, when a user moves in a virtual vehicle in a virtual space, the user does not move while the virtual space is displayed, but moves in the virtual space by driving/operating the vehicle, and a trigger is generated when the user passes a boundary line or a boundary surface by moving in the virtual space, and a display transition is executed to start displaying the real space. Similarly, in this embodiment, the server determines whether the physical condition level of each user is at a danger level or not, and transmits warning information to users who are at the danger level or who may reach the danger level within a predetermined time, but the server may determine the physical condition level of each user at a multiple level (e.g., three levels of normal level, caution level, and danger level) and transmit warning information to users who have reached a predetermined level (e.g., caution level) that does not reach the danger level or who may reach the predetermined level within a predetermined time. When the user turns around, the room door reappears in the user's field of view, and what the user sees through the room door is the real screen outside the room. In other words, in this embodiment, if the current position of the terminal does not pass through the spatial region again and the current field of view covers the virtual entry due to movement, a field of view located within the virtual entry in the current field of view is determined, and the screen within the field of view determined in the acquired image frame is displayed in the virtual entry.

Here, the control unit may determine the timing to execute a human error induction program that attempts to induce mistakes due to frustration (stress) based on the GSR detection signal and heart rate detection signal output by the GSR detection unit and heart rate detector as biometric information detectors, or may select whether to move the virtual body less than intended by the worker when moving it toward the dirt, to make the dirt harder to remove, or to display information that will irritate the worker (stress the worker), or to execute a combination of these.

When the server device determines the character image to be displayed based on the user's status determined from the bio-information, exercise information, and diet information, the server device may transmit to the bio-motion information display device information specifying the character image to be displayed and information specifying the expression and movement of the character to be displayed, rather than transmitting the character image data itself. When the server device determines the character image to be displayed based on the user's status determined from the bio-information/exercise information, the server device may transmit to the bio-motion information display device information specifying the character image to be displayed and information specifying the expression and movement of the character to be displayed, rather than transmitting the character image data itself.

So far, we have explained the operation of displaying a character image according to the user's own current situation and reproducing and displaying a character image showing a past situation. Next, we will explain a bio-motion information processing system that allows the user to know the situation of others through a character image, and the devices that make up said system. Furthermore, if the detected bio-information such as heart rate and blood pressure is a normal value, it is not used to display a character, and only if the value becomes physically abnormal, the bio-information is used as the subject of the character display, thereby displaying a character that has the meaning of a warning or caution about the user or others.

In one example of this embodiment, it is not necessary for the character posture setting unit to use the positions of the user's body parts in real space and a scaling factor to determine the position coordinates in the virtual space of all of the above-mentioned nodes of the virtual character. Instead, the position in the virtual space of a pre-selected selected node among the nodes included in the bone model of the virtual character may be determined using the estimated positions in real space of the user's body parts and a scaling factor, and then the positions of each node included in the bone model of the virtual character other than the selected node may be determined based on the position in virtual space of the selected node thus determined.

In the physical function training method of this embodiment, for example, the movement information conversion process may include a current position measurement process and a training position information generation process, in which the current position measurement process measures position information of the entire or part of the user's body from the acquired movement information, the training position information generation process calculates a current speed from the position information and generates training position information by adding or subtracting a set speed from the current speed, the training movement information in the movement information conversion process includes training position information, and the image information generation process may generate training virtual space image information that reflects the training position information in the virtual space.

The patent document proposes a wrist-worn environmental data measuring device that includes a case to which a wrist band is connected, a temperature sensor for detecting temperature, and a display control means for displaying the temperature on a predetermined area of a display section formed on the surface of the case based on the detection result of the temperature sensor, and the temperature sensor is arranged with its heat-sensitive section protruding from the case. Even if the weather measuring device does not include a wind speed sensor, the activity amount calculation section can calculate the pressure gradient (= pressure difference between two points/distance between two points) between two points (between two weather measuring devices) from the pressure data and estimate the current wind speed from the pressure gradient distribution, so that the activity amount of each user can be corrected according to the estimated current wind speed. Biometric information is, for example, information such as heart rate, pulse rate, respiratory rate, blood sugar level, sweat rate, body temperature, and salinity concentration in blood, and the biological information sensor is a sensor that acquires at least a part of the biological information from these various biological information, and is worn in a position where it can acquire the necessary biological information from the user.

Here, GX, GY, and GZ are digital values (angular velocity information) that correspond to the direction and magnitude of the angular velocity around the X-axis, Y-axis, and Z-axis applied to the sensor module, respectively, and AX, AY, and AZ are digital values (acceleration information) that correspond to the direction and magnitude of the acceleration in the X-axis, Y-axis, and Z-axis directions applied to the sensor module, respectively.

For example, if biometric information about the user of the biometric information transmission device is stored for a certain period of time in the biometric information database of the server device, it is possible to specify a past time within the stored range and extract the biometric information, making it possible to display the past condition of other people using a character image.

When the server device determines the character image to be displayed based on the user's status determined from the bio-information, exercise information, and diet information, the server device may transmit, to the bio-exercise information display device, information specifying the character image to be displayed and information specifying the expression and movement of the character to be displayed, rather than transmitting the character image data itself to be displayed.

In other words, in a system in which the bio-kinetic information transmitting device and the bio-kinetic information display device communicate directly, the bio-kinetic information display device can receive bio-information, exercise information, dietary information, location information, etc. detected by the bio-kinetic information transmitting device, making it possible to display characters based on other people's bio-information, exercise information, and dietary information, or to display characters on a map image using location information.

When the server device determines the character image to be displayed based on the user's situation determined from the bio-information/motion information, the server device may transmit information specifying the character image to be displayed and information specifying the facial expression and movement of the character to be displayed to the bio-motion information display device, rather than transmitting the character image data itself to be displayed. That is, in a system in which the bio-motion information transmitting device and the bio-motion information display device communicate directly with each other, the bio-motion information display device can receive bio-information, motion information, position information, etc. detected by the bio-motion information transmitting device, and thereby display a character based on the bio-information and motion information of another person, or display a character on a map image using position information.

The target pace extracted from the target pace database may be generated based on the most recent running distance and running time from among the training data of running that the user has done in the past, or, if the training data is stored in the target pace database in association with topographical information of the running course (elevation, course shape, etc.) and meteorological information at the time (temperature, wind direction, wind speed, etc.), the target pace may be generated based on the running distance and running time extracted based on various conditions such as the topographical information and meteorological information input by the user.

In addition, in each of the above-mentioned operation modes, when the user temporarily stops exercising (running) by taking a rest, drinking water, or the like, and the detected running speed drops (slows down), the image of the virtual runner may be temporarily stopped, for example, by the user operating the operation unit so that the virtual runner does not increase its running speed (pace up) and move away based on the above-mentioned processing operation. In a step, when it is determined that the pacemaker virtual object has reached the right edge of the display screen (determination of the right edge of the screen display), the calculation circuit replaces the target pace set for that running section with the user's current running pace (actual pace) calculated based on the GPS data, and displays it in the target pace display on the display screen.

For example, when a decrease in the user's pitch is detected, when a slowdown in the user's pace is detected, or at any other time, a moving image may be generated and displayed in which the time from when the virtual runner's foot leaves the ground to when it lands is slightly faster than the user's pitch, i.e., the pitch is slightly faster, to encourage the user to increase their pitch. Also, it has been very difficult to secure personnel who meet the capabilities and conditions required of a pacemaker, namely, a runner with the same or greater running ability as the user and who can understand the user's ability and provide appropriate guidance, but by applying the exercise support device and exercise support method described above, a pacemaker that is optimal for oneself can be easily used.

For example, when the visual virtual agent determines that the user is running, the user's activity data indicates that the user is running, and the activity data can be acquired using a sensor such as a GPS tracker or a step sensor, and the visual virtual agent may be further configured to acquire user-specific data, for example, from the GPS tracker or the step sensor, and thus may be configured to monitor and automatically determine whether the user has stopped running. The distance traveled when not participating in a game may be measured by a terminal device or a pedometer, and the number of games to be played and the distance traveled may be calculated based on the distance traveled and the distance traveled during the game in order to burn a target amount of calories.

In this case, the pitch set for the virtual runner may be, for example, a uniform number of steps faster than the user's pitch, or may be, for example, a certain ratio of steps faster depending on the user's pitch, or may be variably set based on other calculation formulas, conditional formulas, etc. The motion measurement unit may measure the distance traveled using GPS, or may calculate the distance traveled by measuring the number of steps and multiplying the number of steps by the stride length. The management unit may change the size of the character image depending on, for example, the movement history, that is, the distance, movement speed, number of steps, etc. traveled by the user in real space.

In addition, in the above embodiment, in the virtual camera interval determination process, the interval between the outer imaging unit (left) and the outer imaging unit (right) in the marker coordinate system is calculated, and then the position and orientation of either the left virtual camera or the right virtual camera is determined from the position and orientation of the other virtual camera calculated based on the marker recognition result based on that interval. However, in other embodiments, the position and orientation of the outer imaging unit (for example, the average position of the position of the outer imaging unit (right) and the position of the outer imaging unit (left), or the average orientation of the orientation of the outer imaging unit (right) and the orientation of the outer imaging unit (left)) may be calculated based on the position and orientation of the outer imaging unit (left) calculated based on the marker recognition result for the left real world image and the position and orientation of the outer imaging unit (right) calculated based on the marker recognition result for the right real world image, and the positions and/or orientations of the left virtual camera and right virtual camera may be determined based on this. In one example of this embodiment, it is not necessary for the character posture setting unit to use the positions of the user's body parts in real space and a scaling factor to determine the position coordinates in the virtual space of all of the above-mentioned nodes of the virtual character. Instead, the position in the virtual space of a pre-selected selected node among the nodes included in the bone model of the virtual character may be determined using the estimated positions in real space of the user's body parts and a scaling factor, and then the positions of each node included in the bone model of the virtual character other than the selected node may be determined based on the position in virtual space of the selected node thus determined.

Alternatively, the trained model may generate time series data as learning data by having the trainer tag the time series data (point cloud data, photographed images of the trainee) that are the results of measuring the trainee's body movements (comments on how to move body parts in an ideal way, that the trainee should breathe, that the trainee's heart rate is too high and that the trainee should take a break, etc.).

In addition, when the assumed position is calculated from the assumed moving speed of the guide as in the portable mobility support device described in the claims, the heart rate meter 9 is used to determine the assumed moving speed, and if the user's heart rate is higher than the target heart rate, the assumed moving speed of the guide is slowed down, and if it is lower, the assumed moving speed of the guide is accelerated. Here, "based on the GSR detection signal and the heart rate detection signal" means, for example, executing a human error inducement program in response to the change in the numerical value indicated by the GSR detection signal and the heart rate detection signal crossing a predetermined threshold, or setting the specific content of the human error inducement program to be executed in advance in response to the numerical value or change in numerical value indicated by the GSR detection signal and the heart rate detection signal, and executing the content. Exercise intensity is an index that indicates the degree of intensity of a person's exercise state (movement state), and may be, for example, an index corresponding to a person's moving speed, vibration, heart rate, body temperature, amount of sweat, etc., or may be subjective exercise intensity (an index that indicates the intensity of exercise from a psychological aspect of a person).

Here, the game status may be, for example, the status of the user's gameplay, the progress of the game, the status of events in the game, the battle status of the game, the match status of the game, the attack status of enemies appearing in the game, the status of hit judgment in the game, the status of moving objects appearing in the game (characters or hit objects, etc.), the status of the status parameters of the characters or the user (stamina value, level, attack power or defense power, etc.), or the status of the game results, and are various situations represented by, for example, game parameters, etc. For example, the output control unit can output the practice menu selected by the processing unit as recommended information, and more specifically, it can control the display unit (display device) of the feedback device to display a predetermined image, control a force-sensing mechanism (wearable device) attached to the user's body to provide tactile stimulation to the user, or control an audio output unit (audio output device) to output a predetermined sound. Furthermore, if the user is wearing a device capable of presenting a haptic sensation on a hand or the like, the information processing device may instruct the device to present a predetermined haptic sensation when a target virtual object in the virtual space comes into contact with the user's finger, or when the contact state between the target virtual object and the user's finger is maintained.

Physical quantities defined in physics such as Newtonian mechanics and thermodynamics include, for example, force (F = mass x acceleration), momentum (P = mass x velocity), work (W = force F x distance traveled), kinetic energy (K = 1/2 x mass x speed2), heat (J = 4.184 x kinetic energy), velocity (rate of change of position of an organism per unit time), acceleration (rate of change of velocity per unit time), and acceleration rate (jerk: rate of change of acceleration per unit time). Here, the time derivative of angular momentum is torque, and the time derivative of momentum is force, and in order to continuously generate torque and force in a fixed direction, it is necessary to continuously accelerate the rotation speed of a motor or a linear motor, and therefore a method of periodically rotating a rotating body is not suitable for continuously presenting a force sensation in a fixed direction.

The reference operation instruction direction may be calculated by correcting the acceleration indicated by the acceleration data, or may be calculated by correcting the direction of the magnetism detected by the magnetic sensor, or may be calculated by having the user perform a predetermined operation with the terminal device in a specific position (e.g., placed on a table and stationary) and assuming that the x-axis direction and the y-axis direction at the time the predetermined operation is performed are horizontal. The data may include data captured from a sensor (e.g., operably coupled to the frame or otherwise attached to the user), such as an image capture device (e.g., a camera in an inward-facing imaging system and/or an outward-facing imaging system), an audio sensor (e.g., a microphone), an inertial measurement unit, an accelerometer, a compass, a global positioning system unit, a wireless device, or a gyroscope, or data obtained or processed using a remote processing module or a remote data repository, possibly for processing or reading and then passing to the display. The data may include data captured from sensors (e.g., that may be operatively coupled to the frame or otherwise attached to the user), such as image capture devices (e.g., cameras in the inward-facing imaging system and/or the outward-facing imaging system), microphones, inertial measurement units, accelerometers, compasses, global positioning system units, wireless devices, or gyroscopes, or data obtained or processed using a remote processing module or remote data repository, possibly for transmission to the display after such processing or readout.

The virtual environment may be used to simulate a real environment in reality, for example, the virtual environment may include sky, land, or sea, the land may include environmental elements such as a desert or a city, the user may control virtual objects to move in the virtual environment, the virtual objects may be virtual avatars representing the user in the virtual environment, the virtual avatars may be of any form, for example, a human or animal, and the present application is not limited thereto.

In one possible embodiment, the first electronic device is an electronic device capable of displaying a virtual environment, i.e., an electronic device equipped with a game application, and each first electronic device can control at least one virtual object, and multiple first electronic devices can control multiple virtual objects, and the user can control through the first electronic device such that the virtual objects fight in the virtual environment, and this first electronic device may be an electronic device such as a mobile phone, a tablet, a laptop, etc., and this first electronic device may be equipped with a Bluetooth (registered trademark) module used to establish a connection with a Bluetooth (registered trademark) module of another electronic device. In order to solve the above problems, the virtual environment construction device is a virtual environment construction device that constructs a virtual environment for a user to view based on a real environment in which a plurality of objects exist, and includes: a model construction method determination unit that determines a model construction method indicating whether to construct a model of at least one of the plurality of objects as a target object using CG or live-action video based on at least one of the sensing difficulty and CG drawing difficulty of the target object; a state sensing unit that measures the state of the target object in the real environment; an environmental model construction unit that constructs an environmental model for drawing in the virtual environment for static objects that do not change over time according to the model construction method; and a dynamic object construction unit that constructs a model for drawing in the virtual environment for dynamic objects that change over time according to the model construction method and synthesizes it into the environmental model. The virtual object is a virtual object controlled by the first electronic device, the virtual object may acquire at least one virtual item in the virtual environment, the virtual items may be bound to the account of the virtual object, the virtual object may indicate that it has the virtual items, the virtual object may use or discard the virtual items, and the first electronic device transfers the virtual item to the virtual environment by the virtual object discarding the virtual item in the virtual environment.

In the factory calibration module, some of the information required to compute the transformations applied to the images/video may be generated during factory calibration or may be computed based on further information on the actual camera orientation at the site, such as height above the floor or desk, 3D viewpoint, lens field of view, etc. As described above, when the controller is tilted beyond the limits of the virtual camera's range of motion and then the pitch angle of the controller is returned, the pitch rate may be changed according to the pitch angle of the controller according to this changed correspondence until the pitch rate of the virtual camera becomes zero, or the correspondence may be gradually returned to the original correspondence. Also, in the above example, as the pitch angle of the controller falls below the change limit and becomes smaller, the correspondence gradually changes to approach the original correspondence, but instead, the correspondence may be gradually changed to approach the original correspondence according to the rate at which the pitch angle of the controller is reduced.

Short-term feedback may include visual indications of the quality and completion of the repetitions, such as a scoreboard that remembers the number of repetitions, a virtual character that mimics or reflects the player's movements, animation of the virtual character that cheers, sings, plays music, and/or otherwise motivates the player to continue, and physical changes in the virtual reality environment, such as the sun rising in the virtual reality environment, food growing, ice melting, wind blowing, or construction projects (initiated by villagers supported by the player). The exercise evaluation unit may also compare the accumulated exercise calories with the total calories consumed by the user within a predetermined period (e.g., during one day), or may compare the accumulated exercise calories with the carbohydrate calories consumed by the user within a predetermined period. For example, the exercise time setting unit may calculate the target exercise time as described above for a user with a diabetic constitution, using carbohydrate calories as the calories to be consumed, and for other users, may calculate the target exercise time using total calories as the calories to be consumed. The control element module may include one or more of the following features: control and management of camera settings to optimize quality, configuration of other hardware elements, communication between the algorithm module and higher-level code, applications, and user interfaces, and pushing calibration data from the factory to the algorithm element. Total calories and carbohydrate calories may be calculated by the user information acquisition unit based on user information, or may be calculated by another device (e.g., a wearable device, a server, etc.).

A display control device according to one aspect of this embodiment is a display control device that controls the display of a head-mounted display equipped with a display unit capable of displaying a virtual space image representing a virtual space seen in a viewing direction from a virtual viewpoint in the virtual space as a stereoscopic image utilizing binocular parallax, and includes a detection result acquisition means for acquiring a detection result of the orientation of the head-mounted display, a first virtual space image acquisition means for acquiring a new first virtual space image in a viewing direction corresponding to a changed orientation of the head-mounted display based on a change in the orientation of the head-mounted display, a second virtual space image acquisition means for acquiring a new second virtual space image in a viewing direction corresponding to a changed orientation of the head-mounted display based on the change in the orientation of the head-mounted display, and a display control means for displaying the first virtual space image and the second virtual space image. The display control device includes a display control means that executes control to synthesize images and display them on the display unit, and a change means that, when the first virtual space image and the second virtual space image are image-composited and displayed on the display unit, reduces the degree of obscuration of the second virtual space image by the first virtual space image in accordance with the passage of time.When the first virtual space image and the second virtual space image are image-composited and displayed on the display unit, the display control device further includes a means for restricting new acquisition of one of the first virtual space image and the second virtual space image, and the display control means moves one of the virtual space images displayed on the display unit in a movement direction based on the opposite direction to the change in orientation of the head mounted display, based on a change in orientation of the head mounted display, and updates the other of the first virtual space image and the second virtual space image displayed on the display unit to the newly acquired virtual space image. As described above, as one aspect of the above embodiment, an information processing device is configured to include a position information acquisition unit, a time information acquisition unit, a weather information acquisition unit, a natural light source estimation unit, a reality corresponding object information acquisition unit, a photographed image acquisition unit, a virtual photographed image generation unit, a photographed object identification unit, and a storage unit, and acquires position information of a user terminal, acquires current time information, acquires weather information for the position based on the position information of the user terminal, estimates the status of a natural light source at the position of the user terminal based on the acquired position information, time information, and weather information, acquires information on a reality corresponding object in a virtual space that is previously associated with a real object existing around the user terminal based on the position information, and photographs the real space with a camera device equipped in the user terminal. The captured image and shooting direction information obtained in the above are acquired, and a virtual camera is installed for at least one pattern estimated to be in approximately the same position and approximately the same shooting direction as the captured image, and images are taken in a virtual space. A rendering is performed that takes into account the effect of the light source on the corresponding object in reality in the estimated natural light source conditions to generate a virtual captured image, and matching points of visual features between the captured image and the virtual captured image are identified, and objects shown in the captured image are identified based on the degree of matching between the two. Therefore, by using a virtual captured image obtained by rendering in a more realistic condition that reflects the natural light source conditions, matching can be performed under conditions where conditions such as shadows, reflections, and brightness are similar, which makes it possible to improve tracking accuracy.

As described above, as one aspect of the above embodiment, the information processing device is configured to include a location information acquisition unit, a time information acquisition unit, a weather information acquisition unit, a natural light source estimation unit, a reality-corresponding object information acquisition unit, a captured image acquisition unit, a virtual captured image generation unit, a captured object identification unit, and a memory unit, and acquires location information of a user terminal, acquires current time information, acquires weather information for the location based on the location information of the user terminal, estimates the status of natural light sources at the location of the user terminal based on the acquired location information, time information, and weather information, acquires information on reality-corresponding objects in a virtual space that are pre-matched to real objects existing around the user terminal based on the location information, and captures an image of the real space using a camera device equipped in the user terminal. The captured image and shooting direction information obtained in the above are acquired, and a virtual camera is installed for at least one pattern that is estimated to be in approximately the same position and approximately the same shooting direction as the captured image, and images are taken in a virtual space. A rendering is performed that takes into account the effect of the light source on the real corresponding object in the estimated natural light source conditions to generate a virtual captured image, and matching points of visual features between the captured image and the virtual captured image are identified, and the objects shown in the captured image are identified based on the degree of match between the two. Therefore, by using a virtual captured image obtained by rendering in a more realistic condition that reflects the natural light source conditions, matching can be performed under conditions where conditions such as shadows, reflections, and brightness are similar, which makes it possible to improve the accuracy of tracking. The main virtual camera includes a left main virtual camera and a right main virtual camera, and the sub virtual camera includes a left sub virtual camera and a right sub virtual camera, and the steps include updating the field of view of the left main virtual camera and the field of view of the right main virtual camera in response to the movement of the head mounted display, and the steps include generating first left eye field of view image data based on the field of view of the left main virtual camera and virtual space data, and generating first right eye field of view image data based on the field of view of the right main virtual camera and virtual space data, and the steps include identifying the position and orientation of the left sub virtual camera based on the relative positional relationship between the left main virtual camera and the reflective object, and identifying the position and orientation of the right sub virtual camera based on the relative positional relationship between the right main virtual camera and the reflective object, and the steps include generating second left eye field of view image data based on the field of view of the left sub virtual camera and virtual space data, and generating second right eye field of view image data based on the field of view of the right sub virtual camera and virtual space data.

A head mounted display system comprising: a frame configured to be supported on a user's head; a display disposed on the frame, configured to project light associated with a virtual image into the user's eye to display virtual image content to the user, and further configured to transmit light from the environment to the user's eye to provide the user with a view of a portion of the environment; a camera configured to acquire images of the environment at a frame rate, the camera having a field of view and a first frame rate; and processing electronics configured to receive and process images acquired by the camera, the image processing including a step of assessing relative motion between the head mounted display and one or more features in the environment, the assessment of relative motion including determining whether the head mounted display has moved, is moving, or is expected to move relative to one or more features in the environment. determining, and/or determining whether one or more features in the environment have moved, are moving, or are expected to move relative to the head mounted display; determining one or more virtual image content locations in the environment that correspond to locations where renderable virtual image content will appear to the user when the locations appear in the display; comparing one or more virtual image content locations in the environment to a visibility zone that includes at least a portion of the field of view of the camera; and modifying a frame rate of the camera from a first frame rate to a second frame rate and/or adjusting an amount of processing on frames captured by the camera to be processed based on the assessment of relative motion between the head mounted display and one or more features in the environment and the comparison of one or more virtual image content locations in the environment to the visibility zone.

For example, all modules of the virtual camera control system may be included in a user terminal device (e.g., a desktop computer, a notebook computer, a workstation, a PDA, a web pad (particularly, a smart pad), a mobile phone (particularly, a smartphone), etc.) (not shown) that typically requires the use and control of a virtual camera, but some modules may be included in other digital devices (e.g., a three-dimensional content providing server) (not shown) that can communicate with the user terminal device other than the user terminal device. According to an aspect of the present disclosure, an information processing device includes a storage unit that stores a program executed by the information processing device in order to provide a virtual experience to an audience watching a first performance by a performer in real space, a control unit that controls the operation of the information processing device by executing the program, and a display unit that displays the real space. Here, the basic processing of virtual reality and augmented reality will be described. A still image or a video background and object that are real-life images captured by a smartphone camera or the like are mixed and processed (visible/invisible/transparent) based on real map information of a real coordinate system (latitude, longitude, altitude) to display still images or CG animation objects that are 3DCG (three-dimensional computer graphics). The camera or cameras may be an integrated part of several different electronic devices of the user (e.g., smartphone, tablet, PDA, TV), or may be located in other devices of the user (e.g., refrigerator, scale, etc.), or may be located in specific locations in rooms of the user's home (e.g., living room, bedroom, and/or kitchen). The member's terminal may be, for example, a wireless communication device that ensures portability and mobility, including all types of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), etc.

A head-mounted display system configured to project light into a user's eye and display augmented reality image content in the user's field of view to image at least a portion of an environment in front of a user wearing the head-mounted display system, the head-mounted display comprising: a frame configured to be supported on the user's head; a head-mounted display disposed on the frame, the display configured to project light into the user's eye and display virtual image content in the user's field of view, at least a portion of the display being transparent, the head-mounted display being disposed at a location in front of the user's eye such that, when the user wears the frame, the transparent portion transmits light from a portion of the environment in front of the user and the head-mounted display to the user's eye to provide a view of the portion of the environment in front of the user and the head-mounted display; a camera configured to acquire images at a frame rate, the camera having a field of view and a first frame rate; a non-transitory computer-readable medium storing computer-executable instructions; and electronics in communication with the camera and the computer-readable medium, configured to process frames acquired by the camera. and an electronic device configured to execute computer-executable instructions to assess relative motion between the head mounted display and one or more features in the environment, the assessment of the relative motion including determining whether the head mounted display has moved, is moving, or is expected to move relative to one or more features in the environment, and/or determining whether one or more features in the environment have moved, are moving, or are expected to move relative to the head mounted display; determine one or more locations in the environment corresponding to locations where renderable virtual image content would appear to a user if the locations appeared in the display; compare one or more virtual image content locations in the environment to a viewing zone that includes at least a portion of the field of view of the camera; and modify a camera's frame rate from a first frame rate to a second frame rate and/or adjust an amount of processing on frames captured by the camera that are processed based on the assessment of relative motion between the head mounted display and one or more features in the environment and the comparison of one or more virtual image content locations in the environment to the viewing zone.

Here, a plurality of light sources L that illuminate the virtual object are arranged in the virtual space, and the virtual space information includes information on the shape and material (color, texture image) of the virtual object that may be arranged in the virtual space, the position and orientation of the virtual object arranged in the virtual space, and information on the position, direction, color, and intensity of the light source. Note that any type of control of the light source may be used, and for example, the color of the light source may be changed according to the operation input, or the position and irradiation angle of the light source may be changed if the light source is a point light source, or the irradiation direction may be changed if the light source is a parallel light source. For example, a virtual light source may be set using a real image for the left eye, and a virtual object illuminated by the virtual light source set using the real image for the left eye may be captured by the left virtual camera, and a virtual light source may be set using a real image for the right eye, and a virtual object illuminated by the virtual light source set using the real image for the right eye may be captured by the right virtual camera. In this way, when experiencing a virtual apartment room using the MR system, if the position of the light source causes the room to be affected by the shadow of a virtual building outside the room, a simulation that also takes sunlight into account can be performed by changing the color of the three-dimensional model inside the room to take into account the effect. A simplified example of obtaining natural light source information would be to obtain information on the direction of the sun estimated from the date and time and the location information of the user device, and to obtain the color of the sky estimated from weather information to create a hemispherical light source that simply adds diffuse ambient color to all objects in the scene.

The term product as used herein is intended to include a computer program accessible from any computer readable device or storage medium, for example stored on a server coupled to a cloud. For example, in a product design, failure to ensure that parts are not in contact with other parts may lead to malfunction. For example, an object may be an item, a treasure chest, a weapon, armor, furniture, an electrical appliance, or a decoration that is placed in real space. A computer program product includes instructions that, when executed, perform one or more methods as described above. Here, client property information refers to any information related to the client, including the client's product information or configuration information.

Regarding the size of the cap and the size of each of the buttons, if the size of the holes formed in the lower housing are the same, it is preferable to make the cap smaller, and if the sizes of the holes are not the same, they may be about the same size. The cap is integrally formed with a peripheral surface portion formed in a roughly cylindrical shape, a top surface portion that closes the upper side of the peripheral surface portion, and a flange that protrudes outward from the outer periphery of the opening at the lower end side of the peripheral surface portion. The cap is preferably formed using an elastic body.

The target pace extracted from the target pace database may be generated based on the latest running distance and running time of the user's past running training data, or may be generated based on the running distance and running time extracted based on the various conditions such as the topographical information and weather information input by the user when the topographical information (elevation, course shape, etc.) of the running course and the weather information (temperature, wind direction, wind speed, etc.) are associated and stored as the training data in the target pace database. In addition, the target achievement value calculation unit may perform a correction calculation of the remaining distance or remaining time until each user's activity level reaches the target activity level based on the weather data acquired by the weather data acquisition unit from N (one or more) weather measurement devices arranged ahead along each course (each user's exercise route).

In the course selection screen, check boxes are provided to the right of "Course A", "Course B", and "Free Course". The user moves the cursor to one of the check boxes and presses the switch/confirm button, and then moves the cursor to the button and presses the switch/confirm button to select the checked course. In the first and second examples shown in this embodiment, as in the first embodiment described above, GPS data acquired during running by a series of exercise support methods, topographical information and weather information of the running course, etc. may be stored as training data in a target pace database of an information processing device or a network server, and the database may be updated sequentially. Specifically, the simulation processing unit performs a simulation process in which a virtual moving body, which is provided in correspondence with a moving toy and whose running characteristics are set based on the running characteristic data, runs (virtual running) according to the running control data on a virtual course in a virtual space, which is provided in correspondence with the course and whose course characteristics are set based on the course data.

In addition, when there are multiple panoramic images at the same location, different panoramic images may be sent, for example, depending on the actual conditions (season, date, time of day, etc.) at the time the panoramic image is requested, so as to match the actual conditions as closely as possible. However, the position or shape may be varied depending on the time of year (e.g., season), and even gaps may be provided between adjacent HEXes. Similarly, the positions and numbers of live streaming towers and portals located in each of the multiple HEXes may also be varied depending on the time of year (e.g., season).

On the other hand, if each viewer wishes to move the gift at their own time, they may input object generation instructions and move the gift at their own time. In another configuration, the display device's orientation may be temporarily fixed relative to another structure, but people can turn it around or pick it up and move it at their own convenience. Here, players are piled into a campsite on top of a small hill, with an aspiring artist on the left and a nosy neighbor on the right.

A display control device according to one aspect of this embodiment is a display control device that controls the display of a head-mounted display equipped with a display unit capable of displaying a virtual space image representing a virtual space seen in a viewing direction from a virtual viewpoint in the virtual space as a stereoscopic image utilizing binocular parallax, and includes a detection result acquisition means for acquiring a detection result of the orientation of the head-mounted display, a first virtual space image acquisition means for acquiring a new first virtual space image in a viewing direction corresponding to a changed orientation of the head-mounted display based on a change in the orientation of the head-mounted display, a second virtual space image acquisition means for acquiring a new second virtual space image in a viewing direction corresponding to a changed orientation of the head-mounted display based on the change in the orientation of the head-mounted display, and a display control means for displaying the first virtual space image and the second virtual space image. The display control device includes a display control means that executes control to synthesize images and display them on the display unit, and a change means that, when the first virtual space image and the second virtual space image are image-composited and displayed on the display unit, reduces the degree of obscuration of the second virtual space image by the first virtual space image in accordance with the passage of time.When the first virtual space image and the second virtual space image are image-composited and displayed on the display unit, the display control device further includes a means for restricting new acquisition of one of the first virtual space image and the second virtual space image, and the display control means moves one of the virtual space images displayed on the display unit in a movement direction based on the opposite direction to the change in orientation of the head mounted display, based on a change in orientation of the head mounted display, and updates the other of the first virtual space image and the second virtual space image displayed on the display unit to the newly acquired virtual space image. ) In addition, a display control method according to one aspect of the present embodiment is a display control method for controlling the display of a head-mounted display having a display unit capable of displaying a virtual space image representing a virtual space viewed in a viewing direction from a virtual viewpoint in the virtual space as a stereoscopic image utilizing binocular parallax, the display control method includes a detection result acquisition step for acquiring a detection result of the orientation of the head-mounted display, a first acquisition step for acquiring a first virtual space image, a second acquisition step for acquiring a second virtual space image, a display control step for executing control for image synthesis of the first virtual space image and the second virtual space image and displaying the synthesized image on the display unit, and a change step for reducing the degree of occlusion of the second virtual space image by the first virtual space image as time passes, in which, in the second acquisition step, a second virtual space image in a viewing direction corresponding to the changed orientation of the head-mounted display is newly acquired based on a change in the orientation of the head-mounted display, and in the display control step, the second virtual space image displayed on the display unit is updated to the newly acquired second virtual space image based on the change in the orientation of the head-mounted display, and the first virtual space image displayed on the display unit is moved in a moving direction based on the opposite direction of the change in the orientation of the head-mounted display.

Such equipment objects may be, for example, items obtainable in the game (for example, items obtainable from a treasure chest, which is a type of target object), or may be paid items that are given to a user in the game in response to a payment made by the user in the real world. The image frames may be collected from the real world by the terminal through a built-in image collection device or an externally connected image collection device, and the built-in image collection device may be a front camera or a rear camera of the terminal, and the image frames may be collected from a real scene by another device and then transmitted to the terminal. For example, when a skier in the real world encounters an undulating course as shown in the figure, the skier detects a change in the direction of travel (change in acceleration, change in the direction of gravity, change in centrifugal force) using the semicircular canals, and for example, directs the line of sight in a direction that looks into the distance. When the user is located in a field (play field, simulation field, play area) in real space (real world), the position information and direction information may be position information and direction information (position coordinates in a coordinate system with a given position as the origin, and rotation angles around each coordinate axis) in the field. When a user plays on a field (play field, simulation field, play area) or on a movable housing in real space (the real world), the position information and direction information may be position information and direction information (position coordinates in a coordinate system with a given position as the origin, and rotation angles around each coordinate axis) on that field or on a movable housing.

For example, when a given operation input is made to an item at a time, the virtual camera may be moved from its local position (meaning a position in the local coordinate system of the moving object) 342-1 at the time to a given reference position in a given period of time according to a given movement route, when a given operation input is made to an item at a time, the virtual camera may be moved from its local position at the time to a given reference position in a given period of time according to a given movement route, when a given operation input is made to an item at a time, the virtual camera may be moved from its local position at the time to a given reference position in a given period of time according to a given movement route. The camera control data may also include data defining a movement route of a trajectory connecting a given reference position in the local coordinate system of the moving object and a target position set corresponding to an item attached to the moving object, and the first camera control unit may perform a process of moving the virtual camera to the target position along the movement route when a given operation input is made to an item attached to the moving object. Here, the camera control data is defined in a time series from the start point to the end point as shown in the figure, and if the start point data is defined at time and the intermediate point data is defined at time, the virtual camera can be controlled to move from an intermediate point on the movement route to the end point by setting the time position as the playback position. The camera control data also includes data corresponding to a plurality of movement routes with different trajectories for the item, and the first camera control unit may select camera control data corresponding to one of the movement routes based on the position or orientation of the moving object or the value of a predetermined game parameter, and perform the first camera control process using the selected camera control data. On the other hand, the viewpoint of the virtual camera is set to a position shifted from the reference according to the direction Pd of the player character (the direction is, for example, the forward direction or moving direction of the player character, but may also be the attack direction by the player character) based on the point Cph (xcp, ycp, zcp) in the virtual game space determined corresponding to each of the positions Ph (xh, yh, zh) of the player character (i.e., each of the points on the route R).

In addition, in a place such as a slope, the direction of gravity and the normal direction of the outer periphery do not necessarily need to be parallel. In this case, the control unit may guide the user to the slope closest to his current position. Here, the game processing unit performs various game processes such as coin (cost) acceptance processing, various mode setting processing, game progress processing, selection screen setting processing, processing for determining the position and rotation angle (rotation angle around the X, Y or Z axis) of an object (one or more primitive faces), processing for moving an object (motion processing), processing for determining the position of the viewpoint (position of the virtual camera) and the line of sight angle (rotation angle of the virtual camera), processing for placing objects such as map objects in the object space, hit check processing, processing for calculating the game result (achievement, achievement), processing for multiple players to play in a common game space, or game over processing, based on operation data from the operation unit, personal data from the save information storage device, game programs, etc. In addition, in the above embodiment, the angular difference (|60 degrees - 30 degrees| = 30 degrees) between the controller pitch angle at which the virtual camera's pitch rate is 0 (the shooting direction is horizontal) and the controller pitch angle corresponding to the upper limit of the virtual camera's range of motion was the same as the angular difference (|0 degrees - 30 degrees| = 30 degrees) between the controller pitch angle at which the virtual camera's pitch rate is 0 and the controller pitch angle corresponding to the lower limit of the virtual camera's range of motion; however, this embodiment is not limited to this, and the angular difference between the controller pitch angle at which the virtual camera's pitch rate is 0 and the controller pitch angle corresponding to the upper limit of the virtual camera's range of motion may be greater or smaller than the angular difference between the controller pitch angle at which the virtual camera's pitch rate is 0 and the controller pitch angle corresponding to the lower limit of the virtual camera's range of motion. [Another example of acquiring information on the position and tilt of the user's head] In the above description, the information on the position and tilt of the user's head is obtained from a head direction sensor provided on the display device, but this embodiment is not limited to this. For example, the position and tilt angle of the user's head may be detected by capturing an image of the user using a camera placed at a known position in a room where the user is located, and detecting a predetermined point that moves with the user's head, such as the position and orientation of a predetermined marker previously placed on the display device worn by the user. [When bones can be detected] Furthermore, when the control unit can estimate the positions and angles of bones and joints of an object based on image data captured by the imaging unit (without using point cloud information) (methods for estimating bones and joints based on image data are widely known, so detailed explanations are omitted here), the amount of movement of each bone (information on the time change related to the position and direction of each bone) may be estimated based on the estimated position and angle, not based on the amount of movement of the point cloud. In other words, when the HMD 1 is in a forward rotation state and the virtual straight line LC and the operation object Op intersect, if the HMD 1 rotates further in the roll direction by an angle, the display range setting unit slides the display content of the message Ms displayed on the bulletin board Bd by a distance according to the angle in the +ZV direction, and may not change the display content of the message Ms displayed on the bulletin board Bd when the HMD 1 is in a forward rotation state and the virtual straight line LC and the operation object Op intersect, if the HMD 1 rotates further in the reverse roll direction Q-X.

Further, the present invention relates to a visual virtual agent configured to perform an artificial physiological color change response to a user for at least one visual virtual agent state and/or for at least one state of the user, the visual virtual agent being configured to perform a visual response to the user, the at least one visual virtual agent state and/or the state of the user being determined based on user activity data and/or user physiological and psychological data, the visual virtual agent being configured to adapt the visual response to changes in the user activity data and/or the user physiological and psychological data, the adaptation of the visual response at least partially comprising a color change of the visual virtual agent for at least one visual virtual agent state and/or at least one state of the user, the user physiological and psychological data being based on the user's current behavior (non-verbal behavior) and/or current physiological state and/or current mental state (psychological state) and/or medical state. The visual virtual agent is displayed in two or three dimensions on a display device, the visual response includes a posture and/or movement of the visual virtual agent, and the audio response includes a sound, volume, emphasis, and/or accent of the visual virtual agent based on the user's activity data and/or the user's physiology and psychology, and at least one sensor is configured to acquire the user's physiology and psychology parameters by speech recognition, face recognition, pulse measurement, respiration measurement, blood pressure measurement, and/or skin electrical conductivity measurement, and the at least one sensor includes an audio-visual sensor, an activity sensor, a physiological sensor, a biometric sensor, a heart rate sensor, a blood pressure sensor/monitor, a weight scale, a motion sensor, a light sensor, a video sensor, an acoustic sensor, a blood glucose monitor, a blood oxygen saturation monitor, a moisture monitor, a skin temperature gauge/thermometer, a respiratory monitor, an EEG electrode, a bed sensor, an accelerometer, an activity sensor/tracker, a video camera, a depth sensor, a skin electrode activity sensor, a portable global positioning system sensor, and a microphone. The present embodiment further relates to a visual virtual agent configured to perform an artificial physiological color change response to a user for at least one visual virtual agent state and/or for at least one state of the user, the visual virtual agent being configured to perform a visual response to the user, the at least one visual virtual agent state and/or the state of the user being determined based on activity data of the user and/or on physiological and psychological data of the user, the visual virtual agent being configured to adapt the visual response to changes in the activity data of the user and/or in the physiological and psychological data of the user, The adaptation of the visual response includes, at least in part, a color change of the visual virtual agent for at least one visual virtual agent state and/or at least one state of the user, the user's psychophysiological data is based on the user's current behavior (non-verbal behavior) and/or current physiological state and/or current mental state (psychological state) and/or medical state, the visual response includes posture and/or movement of the visual virtual agent, and the audio response includes sound, volume, emphasis, and/or accent of the visual virtual agent based on the user's activity data and/or the user's psychophysiological data.

For example, if the user suddenly starts walking faster or running, the generation unit may change the tempo of the music to an up-tempo, or may change the song itself to a faster tempo and generate music. That is, for example, if a specific piece of music is played at a tempo faster than normal, the time from the start of playback of the music to the first first timing is shorter, and the time between the first timing and the second timing is also shorter.

Here, in a predetermined coordinate system (for example, an X1Y1 coordinate system with the center of the platform as the origin, the long side direction of the platform as the X1 axis direction, and the short side direction as the Y1 axis direction) corresponding to the position on the platform of the board-type controller, an area (right foot area) to which the center of gravity moves when the user lifts the left foot and lowers the right foot in a stepping motion, and an area (left foot area) to which the center of gravity moves when the user lifts the right foot and lowers the left foot in a stepping motion are set. Here, F indicates the number of frames from when the stepping flag is turned on to the current time (the number of times the game processing cycle has been repeated, and the game processing time = 1 frame time), and indicates the number of frames from when the user's previous stepping on the board-type controller is detected to the current time (i.e., the time when the current stepping is detected). Here, the threshold value used in the above step is a value for counting the stepping of the user by detecting that the load that rose once when the user stepped on the board-type controller has fallen, and may be a preset value or a value set by the user's operation. Here, the threshold value used in the above steps is a value for detecting that the user has stepped on the board-type controller, and may be a preset value or a value set by the user's operation. In addition, in the above-mentioned stepping determination, it is determined whether the user has stepped on the board-type controller with their left or right foot, but such a distinction between left and right is not necessary.

Alternatively, the user may move the cursor to the display position of "activity amount" and press the switch/determine button, and then each time the up button or down button is pressed, "activity amount" and "calories burned" are alternately displayed (and the unit of measure is alternately displayed in conjunction with this), so that the user can select the input item. Also, based on the calorie consumption per game, the number of times the game needs to be played (for example, the number of times the game needs to be played again) to consume the target calorie consumption amount or the general calorie intake may be calculated, and the number of times may be displayed on the UI section of the terminal device. The distance traveled when not participating in a game may be measured by the terminal device or a pedometer, and the number of games to be played and the distance traveled, etc. to consume the target calorie amount may be calculated based on the distance traveled and the distance traveled during the game. In the above-mentioned embodiment and modified example, the control section may calculate the remaining energy (calories) that can be consumed that day (=total calories - calories consumed) based on the total calories required for the day and the energy (calories) of the food actually consumed by the user, and provide the user with information indicating the remaining calories that can be consumed. The exercise evaluation unit may also compare the accumulated exercise calories with the total calories ingested by the user within a predetermined period (e.g., one day), or may compare the accumulated exercise calories with the carbohydrate calories ingested by the user within a predetermined period. Determination method using an infrared sensor In the current mainstream HMD, an infrared sensor is used to track the user's position information so that the user can move freely around the virtual space.

A system as described in any one of appendices 1 to 7, in which, in the sensing step, the first processor identifies a specific evaluation value related to the body movements of the first user based on the point cloud data of the first user, and at least one of the first processor, the second processor, or a processor of a computer capable of communicating with the first user's terminal and the second user's terminal identifies a third user who will train together with the first user based on the evaluation value of the first user, and in the presenting step to the second user, the second processor receives information based on the point cloud data of the first user and point cloud data obtained by sensing the body movements of the third user, thereby presenting an image including a first avatar and a second avatar corresponding to the body movements of the third user to the second user. The "Sensing results" item is the results of the trainer's body movements during training measured by the terminal device using a measurement unit (point cloud data), or an image in which the point cloud data is placed in a virtual space and rendered based on the settings of the virtual camera (including an avatar that represents the trainee's body movements as a collection of point cloud data, and auxiliary lines drawn according to the trainee's body movements (the type of body movements presented to the user, such as lateral raises)).

In this case, the method of identifying the physical fitness level may be such that the terminal device or server identifies the physical fitness level of the user based on a learned model or the like when sensing the user's physical movements, or the user (trainer) in charge of instructing the user (trainee) receiving training may identify the trainee's physical fitness level based on the results of sensing. For example, the user may set the manner of moving the body on the terminal device, and while drawing auxiliary lines or the like according to the set manner of moving the body, notify the user of the information on the manner of moving the body (guide the user on how to move the body), and sense the manner of moving the user's body using a measurement unit or the like. In this case, if it is determined that accurate sensing is not being performed by analyzing the acquired bioinformation, the accuracy of sensing can be further improved by changing the measurement parameters, such as changing the measurement wavelength band of the sensor unit or changing the measurement time.

This allows the user to watch the race between his/her robot device and another user's virtual robot on a racing mat in his/her home. On the other hand, walking actions such as jogging are less dangerous than mountain climbing or hiking because the walking action can be stopped at any time when the user feels that his/her physical strength is at its limit. However, if the user can grasp the amount of exercise corresponding to his/her physical strength in advance and perform the walking action, jogging will not damage his/her physical condition, so it is also a good idea for health management. The "posture" may be determined so that if "jogging" is input as the exercise type to be input, the character is displayed in a running posture, and if "walking" is input, the character is displayed in a walking posture. In general, when continuing to walk such as mountain climbing, hiking, or jogging, which consumes physical strength, it is important to make a plan according to the physical strength of the walking performer. After controlling the steps, the processor may cause the HMD to display a screen that prompts the user to place the controller between the user's legs.

The movement of the virtual person, etc. may be any of upward, downward, leftward, rightward, etc., and may be in a direction along the direction of the movement of the upper body, or may be in another direction. In this respect, in this embodiment, when the robot's movement state becomes a predetermined state (high-speed rotational movement, etc.), the displayed objects other than the enemy robot ERB that the user is gazing at become blurred images. Here, orthogonal transformation means a coordinate transformation such as parallel movement, rotational movement, inversion (mirror image) movement, etc., in which the distance between points does not change due to the transformation. The movement may include a speed, acceleration, or position change (rotation, movement from one place to another place, etc.). The application control program described in any one of claims 1 to 4, further causing the control unit to function as a means for moving a moving object that is the target of a second action in an image of the virtual space output to the wearable display, the means for performing the first action stops the movement of the moving object based on the operation of the display as the first action, and the means for outputting an image of the virtual space reflecting the first action stops the movement of the moving object in the image output to the wearable display. Furthermore, in the "flight mode", when the left foot position and the right foot position exceed the front boundary and the position of the tracking sensor attached to the head or the tracking sensor attached to the upper body other than the head moves to a rear position relative to the initial position at the start of the game, that is, when the head or the upper body other than the head is tilted in the opposite direction to the direction in which the legs are extended, the speed and distance of movement of the seat object and the virtual person may be increased to reflect the action. The application control program described in any one of claims 1 to 5, wherein the means for outputting an image of the virtual space in which the first action is reflected displays a predetermined object included in the overlapping portion when the determined display range and the attention range according to the first user's posture overlap, and outputs the predetermined object in a non-display state to the wearable display when there is a predetermined object not included in the overlapping portion.

Conversely, in the case of a relationship between a pet and its master, when the face image of the pet is selected, the face images corresponding to the master and his/her family smile or turn their gaze toward the pet, and the game device can reflect the emotions, consciousness, and relationships of the real world in the virtual world. Next, an explanatory diagram showing an example of a state in which a guide (an electronic pet walking on a sidewalk) has moved out of the range of the camera and is no longer displayed on the display, and the guide (an electronic pet that has moved out of the range of the camera and is no longer displayed on the display) cannot be displayed on the display at all. The game device may select the next face image to be used from the game execution history of the user up to the present time based on the face image as well as the attributes of the subject of the face image, such as age, sex, friendship (family, friends, interactions at work, school, local area, etc.), and ownership of the subject when the subject is a living creature such as a pet. For example, the following reference document discloses a technology that makes it easier to recognize the presence of a virtual object by transitioning the internal state of the virtual object through a specified action of a specified part of the user in a mixed reality presentation device such as a game in which a virtual pet is raised. The information processing device may further include a switching unit (and a purchasing unit) that allows the user to select a pet or character to interact with from among multiple types of pets and characters, either free of charge or for a fee, and causes the selected pet or character to appear in the virtual reality space.

In the embodiment, the goggles worn by the cat are closed to prevent the cat from finding the mouse. In another example, when the first elapsed time reaches the first predetermined time, a character such as a dog may appear and block the cat's view, or the cat may be forced to move backwards and close the curtain. For this reason, a predetermined range of depth values (e.g., 0.9-1.0) is stored in each pixel of the area where the dog object is displayed (the display area of the silhouette image), and a predetermined depth value (e.g., 0) is stored in the area where the dog object is not displayed. For example, the face image acquisition process described in the first embodiment may be executed to collect face images of various animals such as mammals such as dogs, cats, and horses, birds, fish, reptiles, amphibians, and insects. When the dog is not interested in the player at all, the dog ignores the player's touch on the touch panel (normal mode), and when the dog is very interested in the player, the dog licks the point touched by the player (licking mode to be described later). For example, a color image of a dog object (which may or may not be a stereoscopic image) may be displayed in one area of the screen, and a silhouette image of the dog object may be displayed in the other area of the screen.

In addition, in the case of a cat object capture event, the state detection processing unit determines that the player's position in the real space indicated by the player state (specifically, the position of the center of gravity or the position of a predetermined part such as the hands or head) has reached the end position on the movement path R, and also determines that the player's posture is crouched and that the positions of both hands in the real space have a predetermined positional relationship with the real space position of the object to be captured (i.e., when contact is determined by contact determination), and determines that the player state is a state in which the cat object capture event has occurred. On the other hand, as with the player falling state, the state detection processing unit may change the above judgment criteria and judgment conditions according to the game environment, such as the player's level (e.g., whether the player is an advanced player or a beginner) and the player's attributes (e.g., gender, age, height), or according to the progress of the game, such as when the player's movement speed is above a certain level, when the time limit is approaching, or when it is taking a long time to capture the cat object. For example, if the pseudo-moving object is an animal object such as a cat, the movement/motion processing unit performs calculations on the movement information and motion information so that the pseudo-moving object moves in a position different from the position where it is placed in the real space, or moves around a different area, and returns to the position where the pseudo-moving object is placed in the real space at a predetermined timing, and becomes in a state similar to the state where it is placed in the real space. For example, in order to obtain facial images of various animals such as mammals such as dogs, cats, and horses, birds, fish, reptiles, amphibians, and insects, the facial image acquisition process described in the first embodiment above may be executed to collect the facial images. The target object may be a human, dog, cat, or other living thing, or may be an inanimate object such as a robot.

An information processing method executed by a computer for providing a virtual space to a user via a head mounted display, comprising: a step of controlling a virtual camera that defines a field of view image from a first character object in a virtual space including a first character object associated with a first user and a second character object associated with a second user; and a position control step of setting the position of the virtual camera to a height based on the first character object and setting the second character object according to the size of the second character object when the virtual space is set to a normal mode in which the first user and the second user share a virtual experience in the virtual space via the first character object and the second character object, respectively, before a predetermined content is played back, and controlling the height of the virtual camera and the eye height of the second character object to have a positional relationship within a certain range in the height direction in the virtual space when the virtual space is set to a viewing mode in which a predetermined content is being played back.

An information processing method executed by a computer for providing a virtual space to a first user via a first head mounted display (HMD device), comprising the steps of: generating virtual space data defining a virtual space including a first player character associated with the first user, a second player character associated with the second user, and a virtual camera defining a field of view image provided to the first head mounted display; and setting the transparency of the second player character included in the field of view image, the second player character including a first part (e.g., head and hands) whose movement in the virtual space is controlled according to the movement of the second user, and a second part (e.g., torso and arms) displayed in association with the first part, and in the setting step, the transparency of at least a part of the second part is set to be greater than the transparency of the first part.

Here, it is described that movies and games can be enjoyed together in a virtual space, and that the layout of a house can be reproduced three-dimensionally in the virtual space, but these are movies and games enjoyed together in a pre-set virtual space, and are not intended to reproduce the experience of enjoying a game, especially a competitive game, together at one's own or a friend's home, and although it is possible to reproduce a situation as if one were back at home by setting the layout oneself, it is not possible to reproduce an experience with a friend in that space. The virtual user may be an avatar image including the whole body, upper body, or face, etc. displayed in the VR space (a part or all of one's appearance is visualized as an avatar image), or only an image of a virtual hand or controller may be displayed in the VR space (one's appearance is not visualized as an avatar image). The online game image may further include various other objects, such as game characters such as NPCs (non-player characters) that appear in the online game, scenery such as buildings and trees, furniture and interior walls in a room, items lying on the ground, parts of the body (hands and feet) of the avatar (oneself) operated by the user, and objects held by the avatar in its hands (guns and swords). Furthermore, if the detected biometric information such as heart rate and blood pressure is normal, it is not used to display the character, and only if the biometric information is abnormal, the biometric information is used as the subject of the character display, thereby displaying a character that is intended to warn or caution the user or others. [Bone model adjustment] The information processing device of this embodiment may also control the display position of the character's hands and feet as viewed from the character's head to match the position (angle from the viewpoint) when the user views the user's own hands and feet in real space, by changing the arm length relative to the height of the bone model of the virtual character used.

The virtual mirror control and virtual system encompasses a broad combination of interfaces, including playback of recorded footage (automatic, manual), progressive effects (color editing of items, augmented reality features, game features, background changes, 3D effects, lighting effects, etc.), screen modes (orientation, full size or split screen), sharing techniques and methods for sharing experiences with friends and experts in real time and on the spot, mobile user applications for controlling display and sharing (e.g., sharing personal footage and images captured from the virtual mirror or the user's mobile device), retail or enterprise remotes for general management (providing real-time retail information about the user to advanced control and analytical functions), and modules for teaching the user how to operate the mirror (animation, video, audio, visual guidance, etc.). In one embodiment, the mirror can be adapted to exhibit pre-determined characteristics based on information about the user's familiarity with the system, for example, based on whether the user is a first-time user, based on the number of user footage in the system, based on when the user last requested the system to enable instructions, etc., provide relatively simple functions such as voice assistance, provide remote expert assistance, etc. The periscope disclosed in this patent document synthesizes a corresponding virtual image with the captured image of the object to be viewed, making it possible to provide the user with a highly realistic synthetic image with accurate scale and positional relationship on the terrain of buildings and the like, without forcing the user to alternate between imagining the actual local scenery and a virtual image of ruins or the like. In this way, when a first-person viewpoint image is displayed on the monitor and the user wearing the HMD turns his head downward, the inside of the avatar object is not drawn, but the mirror object placed in the virtual space displays the overall appearance of the user's avatar object. Here, data on the three-dimensional shape of the heart is generated using CG from images such as MRI (Magnetic Resonance Imaging), and geometric consistency with the position of the endoscope is acquired in advance, and the three-dimensional shape of the CG and the virtual shape are placed in a positionally consistent state in the same three-dimensional space.

An information processing method executed by a computer in an information processing system including a first user terminal having a first head-mounted device worn on the head of a first user and a second user terminal having a second head-mounted device worn on the head of a second user, the information processing method including a step of determining a relative positional relationship between a first avatar associated with the first user and a second avatar associated with the second user, a step of acquiring first virtual space data defining a first virtual space including the first avatar and the second avatar, a step of acquiring second virtual space data defining a second virtual space including the first avatar and the second avatar, and a step of determining a position of the first head-mounted device based on the movement of the first head-mounted device and the first virtual space data. The method includes a step of updating first field of view image data showing a first field of view image displayed on the head-mounted device, and a step of updating second field of view image data showing a second field of view image displayed on the second head-mounted device based on the movement of the second head-mounted device and the second virtual space data, in which in the first virtual space, the relative positional relationship between the first avatar and the second avatar is maintained, and the distance between the center position of the first virtual space and the first avatar is smaller than the distance between the center position of the first virtual space and the second avatar, and in the second virtual space, the relative positional relationship between the first avatar and the second avatar is maintained, and the distance between the center position of the second virtual space and the second avatar is smaller than the distance between the center position of the second virtual space and the first avatar.

The virtual environment may be used to simulate a real environment in reality, for example, the virtual environment may include sky, land, or sea, the land may include environmental elements such as a desert or a city, the user may control virtual objects to move in the virtual environment, the virtual objects may be virtual avatars representing the user in the virtual environment, the virtual avatars may be of any form, for example, a human or animal, and the present application is not limited thereto.

A method for transforming a virtual object located in a three-dimensional space, the method being performed under control of an augmented reality system comprising computer hardware and a user input device, the AR system being configured to enable user interaction with a virtual object within the user's kinesthetic field of view, the FOR including a portion of the user's surrounding environment perceivable by the user via the AR system, the virtual object being presented via the AR system for display to the user, the method including: determining a group of virtual objects within the user's FOR; receiving a selection of a target virtual object within the group of virtual objects within the user's FOR; calculating a distance to the target virtual object; determining a multiplier based at least in part on the distance to the target virtual object; receiving a first movement of the user input device; calculating a second movement of the target virtual object, the second movement based at least in part on the first movement and the multiplier; and moving the target virtual object by an amount based at least in part on the second movement.

Similarly, the hardware for the video playback apparatus may be a conventional computing device operating under suitable software instructions, such as a playback processor (e.g., a CPU operating under suitable software instructions) configured to acquire a first video recording of a video game play session including a sequence of video images, configured (e.g., by suitable software instructions) to acquire a second video recording generated by encoding a sequence of depth buffer values, and configured to acquire a sequence of in-game virtual camera positions associated with at least one of the acquired video recordings, and configured to provide the in-game virtual camera positions to an event analyzer and to generate statistical the analysis processor (e.g., a CPU operating under suitable software instructions) configured (e.g., by suitable software instructions) to obtain data representative of statistically significant in-game events and in-game event locations; a position calculation processor (e.g., a CPU operating under suitable software instructions) configured to calculate a location in a current video image of the first video recording that corresponds to the in-game event location responsive to an associated in-game virtual camera position and a decoded depth buffer value; and an augmentation processor (e.g., a CPU and/or GPU operating under suitable software instructions) configured to augment the current video image with a graphical representation of the statistically significant in-game event responsive to the calculated location. A mapping method may thus include the steps of generating at least a portion of a first map including a graphical representation of some or all of the mapping points of at least a first map data set for display with a first video recording; generating at least a portion of a second map including a graphical representation of some or all of the mapping points of at least a second map data set associated with a different video recording of a different sequence of video images output by the same video game title and sharing the same in-game coordinate system; displaying at least a portion of the first map during playback of the first video recording, the displayed portion including at least a current virtual camera position associated with the displayed video images; and displaying at least a portion of the second map during playback of the first video recording if a respective portion of the second map is within a predetermined range of the current virtual camera position of the in-game coordinate system. That is, the CPU may acquire data for outputting information, move the output source and/or the virtual observation point in a virtual space in which an output source that outputs this information and a virtual observation point that virtually observes the information virtually output from this output source are respectively set, and use a value that limits the magnitude of the speed of the output source and/or the speed of the virtual observation point to apply an effect using the limited value to the information that is virtually observed at the virtual observation point when the information is output from the output source, generate modified information, and output this modified information.

The virtual environment may be used to simulate a real environment in reality, for example, the virtual environment may include sky, land, or sea, and the land may include environmental elements such as deserts and cities, and the user may control virtual objects to move in the virtual environment, which may be virtual avatars representing the user in the virtual environment, and the virtual avatars may be any form, such as humans or animals, and the present application is not limited thereto. The use of multiple labelers (child and caregiver) providing several signals can reduce errors, and a module measuring the rationality of the labels (using the agreement level of the human labelers in conjunction with machine learning algorithms) can decide whether further expert labelers should determine the truth value of the representation, whether the human input should be taken as the truth value, or whether the data should be discarded. In both cases, asking humans to manually correct the labels frame by frame is infeasible, and therefore these embodiments may take an active learning perspective, which relies on various heuristics to select the data to be used in these queries. In the above embodiment, an example was described in which the angle of view of the virtual camera in the initial field of view is an angle of view in which the image field of view is narrower than the human field of view, but the angle of view of the virtual camera in the initial field of view may be an angle of view in which the image field of view is the same width as the human field of view. Specifically, the "predetermined criteria" may include, for example, some or all of whether the motion is executable by a human, whether the motion is executable by a non-human monster or the like, and whether the motion is executable by an artificial object other than a virtual living being.

For example, an icon representing an avatar may be displayed at the current position of the avatar on a plan view of a golf course as shown in the figure. Also, for example, when a user other than the Oni (child) finds the avatar of the Oni user, he or she can press the lower part of the touchpad to make his or her avatar run away from the other user's avatar. For example, when a user who is the Oni finds the avatar of another user, he or she can press the upper part of the touchpad to make his or her avatar run toward the other user's avatar.

The use of multiple labelers (children and caregivers) giving several signals can reduce errors, and a module measuring the rationality of the labels (using the agreement level of the human labelers in conjunction with a machine-learned algorithm) can determine whether a further expert labeler should determine the truth value of the representation, whether the human input should be taken as the truth value, or whether the data should be discarded. If the child object moves from a position farther away from the user than a predetermined distance as shown in the figure to a position less than a predetermined distance as shown in the figure, i.e., if the child object approaches and comes within a range where the user can touch it, the child object is displayed on the screen of the flat image display device. For example, in a case where a child object and a furniture object are displayed on the screen of the stereoscopic image display device, when only the child object is operable (when selected), the child object may be displayed in red on the screen of the flat image display device, and the other objects may be displayed in gray. Therefore, the image displayed in the flat image display area (an image of a virtual space including a child object) is an image of approximately the same size as the image displayed on the stereoscopic image display device (an image of a virtual space including a child object and a furniture object), and is an image of approximately the same range of the virtual space. Alternatively, the item may be pre-existing at a specific position on the screen of the stereoscopic image display device (a specific position in the virtual space), and the user may touch the item image displayed on the flat image display device to hand it over to the child object.

In addition, at least one of the system software, portal software, and game application may not be stored in advance at the time of sale of the information processing device, but may be provided by downloading via a network, and may be updated to the latest version by a network update.

The resolution of the eye camera may be set to high resolution when the user is wearing the wearable device so that a high resolution photograph of the user's entire face is available for use in generating a model of the user's face. In addition, in a ride attraction for children, a composite image (composite video) of a child riding on the ride cabinet may be generated and printed as a photo to take home or distribute. In addition, for example, the camera image may be an image (e.g., a landscape photo) stored in an external data storage memory of the game device. In addition, the map image is image data representing the area where the panoramic photography was performed, and may be a photograph or CG. The "predetermined texture" may be a monochromatic image (e.g., a black image), or may be a photograph or CG image representing the ground, floor, or surface of the earth.

The online game image may further include various other objects such as game characters such as NPCs (non-player characters) that appear in the online game, scenery such as buildings and trees, furniture in a room and interior walls, items lying on the ground, parts of the body (hands and feet) of the avatar (the user himself) operated by the user, and objects (guns and swords) held by the avatar. In addition, when a selected virtual space image related to a scenery seen when driving on a road such as a mountain or a seaside is displayed, if a turning virtual space image related to a seaside scenery is displayed while a selected virtual space image related to a mountain scenery is displayed, the user feels uncomfortable. In addition, when the user terminal captures the surrounding scenery and displays it on the display unit, the virtual object image indicated by the image data may be displayed on the display unit based on the object position information received from the distribution device. In addition, since the operator is not basically performing a shot operation when this camera control function is activated, sound corresponding to the displayed scenery (view) may be output according to the operation of the controller. In addition, in the above embodiment, an example was shown in which a virtual character having a human form was photographed singing and dancing to music, but the subject object could also be an animal, a plant, a landscape, a vehicle, etc.

In the above embodiment, the left and right positions of the left and right fist objects in the virtual space are controlled in response to the operation using the left and right controllers, and the vibration changes temporarily in response to the operation. However, the left and right fist objects may be configured to be controllable in the vertical and/or front-back positions in the virtual space. For example, in the above game example, the arms of the first player object are extended to control the movement during movement, but this embodiment may be applied to a game in which the other five parts of the player object (e.g., legs) extend or an object held by the player object (whip object, bellows object, etc.) extend. In the above embodiment, the virtual microphone is associated with the player object, and the virtual sound source is associated with the non-player object, but this embodiment is not limited to this. For example, the virtual sound source may be associated with the player object (i.e., the position and speed of the virtual sound source may be controllable by the user). In this case, the door in the real space may be fitted with a key that can be controlled to be unlocked and locked from the server device, and the server device may control the door to be locked when a player belonging to a group that is not allowed to pass through that door approaches that door. In this case, the vertical positions of the left fist object and/or right fist object in the virtual space may be controllable according to the vertical movement and/or pitch posture of the left controller and/or right controller in the real space, and the vibration may temporarily change according to that operation.

In the shared mode, when the surgeon specifies a position on the virtual object with his/her finger, the specified position information may be acquired by the specified position information acquisition unit, and the display control unit of the participant's HMD may display an image of an arrow representing the specified position on the virtual object displayed on the participant's display based on the position information acquired by the specified position information acquisition unit. Here, for example, when the surgeon stands in front of the virtual object, the participant stands opposite the surgeon, the participant stands to the right of the surgeon, and the participant stands to the left of the surgeon, the participant observes the back of the virtual object of the head, the participant observes the left side of the virtual object of the head, and the participant observes the left side of the virtual object of the head. In addition, in this embodiment, the display of the virtual object of the participant's HMD is changed in conjunction with the change in the display of the virtual object of the surgeon's HMD as described above, but the display change of the virtual object of each participant's HMD may be performed independently in each HMD without such a linked display. Because the operator's position and the position of the hand for manipulating the virtual object are measured, it is possible to manipulate the virtual object, but the stereo image observed by the participant is an image of the participant with the background removed and an image of the VR space made up of the virtual object, and does not contain information about the participant's real space. Next, when the surgeon instructs the system to switch to normal mode using the switching controller while in the shared mode, the system switches to normal mode, and the display of the participant's HMD displays a virtual object based on the positional relationship between each participant and the marker, i.e., a virtual object oriented according to the position of each participant.

In addition, when, for example, a first finger determined to be facing ventrally and a second finger determined to be facing dorsally are both in contact with the target virtual object, and the distance between the coordinate value (x1, y1, z1) of the fingertip in the XYZ coordinate system of the virtual space of the fingertip and the coordinate value (x2, y2, z2) of the fingertip in the XYZ coordinate system of the virtual space of the fingertip is below a predetermined threshold value (at this time, the part of the target virtual object corresponding to the finger facing dorsally is deformed to be located behind the finger, but the position comparison in the Z-axis direction with the finger facing ventrally is performed using the coordinate value before deformation, so the finger is in a semi-transparent composite and drawn state), the information processing device may determine that the target virtual object is virtually pinched between these fingers at the position of the finger (a predetermined range including (x1, y1) and (x2, y2), for example the range of the smallest circle including these), and execute a predetermined process. Furthermore, in one example of this embodiment, when it is determined that the coordinates of the fingertip of the dorsal finger satisfy the above contact condition, the information processing device compares the Z-axis value (Zj(Xf, Yf)) of the target virtual object at the position of the position coordinate (Xf, Yf) seen from the user's side of the finger with the Z-axis value of the position coordinate of the finger, and if Zth>Zf-Zj(Xf, Yf)>0, it deforms a part of the target virtual object so that the part of the target virtual object corresponding to the position of the finger is located farther away from the position of the user's finger, and if Zf-Zj(Xf, Yf)≧Zth, it may translate the position of the entire target virtual object in the Z-axis direction to set the part of the target virtual object corresponding to the finger farther away from the finger. That is, the CPU 31 may acquire data for outputting information, move the output source and/or the virtual observation point in a virtual space in which an output source for outputting the information and a virtual observation point for virtually observing the information virtually output from the output source are respectively set, and generate correction information by adding an effect using a limited value to the information, which is virtually observed at the virtual observation point when the information is output from the output source, using a value with a limit on the magnitude of the speed of the output source and/or the speed of the virtual observation point, and output the correction information. Also, in the second operation mode, when the player moves the position pointed to by the controller (position of the aiming cursor) while pressing the B button (operation button) of the controller, when the position of the aiming cursor enters the peripheral area of the game image (for example, or the peripheral area outside the displayed image) or when the B button is pressed with the aiming cursor in the peripheral area, the direction of the virtual camera is changed in such a direction that the position of the virtual game space corresponding to the position of the aiming cursor is displayed on the display screen of the monitor (at this time, the viewpoint position of the virtual camera does not need to be changed). The system would preferably be configured to know when a user is holding an item and understand the type of interaction the user is having with the item.

Here, as described above, the exercise support device according to this embodiment is composed of a single device, display glasses, and the display unit has a transparent display surface such as a transparent display panel or transparent glass, so that the user can see the surrounding scenery through the display unit by simply wearing it on the head, and the virtual runner is displayed on the display surface. For example, if the height of the photographer of the shared image and the height of the user who selected the shared image are close, even if the viewpoint of the user is directly moved to the shooting location, the scenery seen after the movement is considered to be closer to the photographer than between users with a large height difference such as a child and an adult, so the viewpoint may be moved directly to the shooting location depending on the combination of the photographer and the user who wants to move. For example, in the above explanation, the shooting space is a real space, and the shooting image is obtained by shooting a real scene, but this is not limited to this, and the shooting image may be an image showing the virtual space as seen from a shooting position placed in the virtual space. The display image in which the user moves in the virtual space may be an image that moves the scenery in the virtual space, or an image with a video effect that evokes movement. In the above explanation, the captured image is an image that includes scenery within a certain shooting range, but it is also possible to use a panoramic image obtained by capturing a wider range using an omnidirectional camera, for example.

In this example of the present embodiment, objects in the real space that are within the reach of the user's hand are reflected as virtual objects in the virtual space, while objects in the real space that are out of the reach of the user's hand are moved or omitted from being displayed as virtual objects, allowing for a highly flexible display. The biokinetic information display device described in the system configuration of the figure may be a device that displays a character image on the display unit that shows the situation of a certain person (a user of the biokinetic information transmission device who is a stranger to the user of the biokinetic information display device) based on information received from the server device. The biokinetic information display device described in the system configuration of the figure may be a device that displays a character image on the display unit that shows the situation of a certain person (a user of the biokinetic information transmission device who is a stranger to the user of the biokinetic information display device) based on information received from the server device. In other words, when the player is located 1 meter away from the position of the input unit, the hand area of the reflected light image will be brighter (the amount of light received will be greater) than when the player is located 2 meters away from the position. The differences between the figure and the flowchart shown in the figure are the generation of a three-dimensional virtual screen of the steps, the projection of a selected image of the steps onto the three-dimensional virtual screen, and the placement of a virtual camera and a virtual water surface within the three-dimensional virtual screen of the steps.

After a certain period of use, the patient and his/her caregiver may review the activities stored from the patient's perspective (with reference to the interactions of others with each other and towards the patient) and the patient's reactions to those situations in the form of behavioral and physiological data (e.g., eye contact made, expressions imitated, etc.). The other person detection unit refers to the captured image output from the imaging device, and detects when a person other than the user wearing the HMD on his/her head appears in the captured image. For example, the other person detection unit detects that a person other than the user has appeared in a position near the user when the captured image changes from a state in which no person other than the user is captured in the captured image to a state in which a person other than the user is captured in the captured image. When the character on the screen is clicked, the other person information is displayed in a separate window, such as the "body shape change", goal achievement rate, and exercise implementation status of the other person. The other person detection unit may detect people captured in the captured image using a known contour detection technique.

In the first and second embodiments, an example was described in which an incentive was given to a virtual user simply by being in the VR space, but a larger incentive may be given to a virtual user who actively participates in activities carried out in the VR space. In the above example, the display mode of the player icon in the map image was different between the part visible from the virtual camera 2 and the part invisible to the camera. In addition, the invisible part (hidden part) may not be displayed (however, in this case, since it is difficult to grasp the position of the player icon when the whole is hidden, it is preferable to change the display mode as described above while always displaying the entire image of the player icon). Here, the basic processing of virtual reality and augmented reality is explained. Still images and video objects that are real-life images taken with a smartphone camera or the like are mixed with objects of 3DCG (three-dimensional computer graphics) still images and CG animations based on real map information of the real coordinate system (latitude, longitude, altitude) and displayed after being mixed (visible/invisible/transparent). The virtual image generation unit is basically the same as that in the embodiment, but unlike the embodiment in which the game output image generated by the game execution unit is fitted into the display unit of the television receiver included in the 3D model, the embodiment is different in that the video output of a stationary game console delivered from a game video delivery unit is fitted into the display unit of the television receiver included in the 3D model.

In the above-described embodiment, a determination is made as to whether an event should be generated based on the approach of the viewpoint to a preset area or the distance between the viewpoint and the AR character, and the behavioral information is registered in the behavior list, but the implementation of this embodiment is not limited to this. For example, a basic volume (basic volume) is set for each sound source, and the game device may calculate the volume according to the distance by multiplying the basic volume by a coefficient (a coefficient that varies in the range of 0 to 1.0 depending on the distance) according to the distance between the sound source and the microphone used for volume calculation.

A video etc. generated based on movement information received from a first sensor (e.g., an inertial sensor) worn by the performer, position information received from a second sensor (e.g., an optical sensor) worn by the performer, body part position information calculated based on the movement information, body part position information calculated based on the position information, body part position information converted from coordinates in physical space to coordinates in virtual space, body part position information converted from coordinates in physical space to coordinates in virtual space, and body part position information converted to coordinates in virtual space. The sensor section may include sensors such as various types of cameras and/or microphones, and at least one processor that processes information obtained by such sensors. A process of drawing the face of an avatar based on measurement data (coordinates of each of multiple parts of the face) of the performer's body acquired by the sensor unit; A process of drawing an avatar at coordinates determined based on body position information calculated by the body position information calculation unit; A process of drawing a part corresponding to each first sensor at coordinates determined based on position information of the first sensor calculated by the part position information calculation unit. Furthermore, generation of such videos can be realized by using various rendering techniques well known to those skilled in the art. Note that in the route of the embodiment, the route of the embodiment, and the route of the embodiment, the terminal device of each viewer does not display a video generated by the distribution device, but can display a video generated (drawn) by the terminal device of the viewer itself (this method is sometimes called "client rendering" because the terminal device of the viewer (client) performs the rendering (drawing)). Here, we explain the basic process of virtual reality and augmented reality. Real-life still images and video backgrounds and objects captured with a smartphone camera or similar device are mixed with 3DCG (three-dimensional computer graphics) still images and CG animation objects based on real map information in a real coordinate system (latitude, longitude, altitude) and displayed as a blend (visible/invisible/transparent).

Short-term feedback may include visual indications of the quality and completion of a repetition, such as a scoreboard that keeps track of the number of repetitions, a virtual character that mimics or mirrors the player's movements, animation of the virtual character that cheers, sings, plays music, and/or otherwise motivates the player to continue, and physical changes in the virtual reality environment, such as the sun rising in the virtual reality environment, food growing, ice melting, wind blowing, or construction projects (initiated by villagers supported by the player). Note that, although a game machine as a home game machine has been described as an example of a computer that functions the game software according to this embodiment, the game machine may be a so-called portable game machine, and further, it may not be a device dedicated to games, but may also be a device that can play general music and video storage media, and is not limited to this, and the computer may be, for example, a personal computer, a mobile phone, etc., that is, any device that can function the game software. In this disclosure, a smartphone is given as an example of a device used in the wearing system, but in this disclosure, a mobile phone, PHS, game machine, watch, music player, car navigation system, television, heart rate monitor, pulse wave meter, blood flow meter, or any other device or any combination of these may be used instead of or together with the smartphone. Alternatively, when a lesson in some movement is taken using footwear (e.g., walking lesson, running lesson, dance lesson, etc.), the generation unit may score whether the footwear is suitable for the lesson based on the movement of the footwear, and generate music according to the score. The generation of music according to the movement of the footwear is realized, for example, by storing sounds to be output in advance in a storage unit, and when the movement of the footwear shows an ideal way of walking or an ideal movement for a certain action, selecting a sound to play a beautiful melody and generating audio information.

The reference virtual plane may be defined as "a plane in a three-dimensional virtual space that a user viewing a stereoscopic display feels exists at a position with a predetermined offset from the display surface of the display device."

Such feedback may be discrete (i.e., indicative of a specific event) or may change over time (e.g., a score such as a "current interest level" score indicating to the patient the level of interest of his/her conversation partner). For example, the game progression unit may award the user with an item that is pre-associated with the scenario played, or may award an item determined based on the play content of the scenario (time required to complete the quest, intimacy level obtained, whether the good option was selected, etc.). The game progression unit may be configured to increase the intimacy level as the play content of the quest (time required, distance traveled, number of items obtained, degree of joy of the character, rarity of the acquired target, etc.) improves. Furthermore, the game progression unit may increment the intimacy level between the protagonist and the character based on the completion of the quest. In one embodiment, the parental review system is a mobile application running on the same phone as the social behavior processing program that performs the computationally intensive emotion recognition. Conversely, the game progression unit may decrease the intimacy level if the user reaches a bad end in the scenario.

Modifications In the above-described embodiment, an example was given of a game in which a target object can be destroyed when an operation is performed on a first controller provided on an operation unit of a first game device in a state in which the target object and the first mark overlap (an example of a case in which the positional relationship between the display object and the first designated position satisfies the third condition), or when an operation is performed on a second controller provided on an operation unit of a second game device in a state in which the target object and the second mark overlap (an example of a case in which the positional relationship between the display object and the second designated position satisfies the third condition). A game may be played in which, when an operation is continued with a first controller provided on a control unit of a first game device in a state where the first object and the second mark overlap (an example of a case where the positional relationship between the displayed object and the second designated position satisfies the third condition), or when an operation is continued with a second controller provided on a control unit of a second game device in a state where the first object and the second mark overlap (an example of a case where the positional relationship between the displayed object and the second designated position satisfies the third condition), the first object can be moved to follow the movement of the first mark or the second mark, or a moving target object can be stopped. More generally, in UCL color video, N bits of depth data (N is typically 16) can be encoded into M bits (M is typically 8) per color space descriptor channel, typically M<P, where P is the native bit depth of each channel in the host data format, and M is preferably 1 bit, more preferably 2 bits less than P. In the game system, a first operation for specifying a position in the image may be performed by operating an operator capable of specifying a direction, such as a cross key, and a second operation for specifying a position in the image may be performed by contacting a touch panel display with a first touch pen. The process for identifying facial expressions or facial actions may include the following fragments: face detection and/or reference point face tracking face and reference size and posture alignment, illumination normalization preprocessing techniques histogram of gradient feature extraction in a region of interest for the face classification of facial expressions via a logistic regression classifier trained on image data, and filtering of the resulting classification. In a method for synthesizing vibration components by rotating two eccentric vibrators in opposite rotation directions, when a high-speed rotation speed ω1 (high frequency f1) at the operating point and a low-speed rotation speed ω2 (low frequency f2) at the operating point are presented alternately every 180° in phase, the haptic illusion intensity (II) is proportional to the logarithm of the acceleration/deceleration ratio Δf/ of the frequency, which is the rotation speed of the eccentric rotor.

In addition, in the above embodiment, in the virtual camera interval determination process, the interval between the outer imaging unit (left) and the outer imaging unit (right) in the marker coordinate system is calculated, and then the position and orientation of either the left virtual camera or the right virtual camera is determined from the position and orientation of the other virtual camera calculated based on the marker recognition result based on that interval. However, in other embodiments, the position and orientation of the outer imaging unit (for example, the average position of the position of the outer imaging unit (right) and the position of the outer imaging unit (left), or the average orientation of the orientation of the outer imaging unit (right) and the orientation of the outer imaging unit (left)) may be calculated based on the position and orientation of the outer imaging unit (left) calculated based on the marker recognition result for the left real world image, and the position and orientation of the outer imaging unit (right) calculated based on the marker recognition result for the right real world image, and the positions and/or orientations of the left virtual camera and right virtual camera may be determined based on the calculated position and orientation. In another example, a real world image may be photographed by a first virtual camera (hereinafter referred to as the real world drawing camera) and rendered to form a first rendering image, and a virtual object may be photographed by a second virtual camera (hereinafter referred to as the virtual world drawing camera) and rendered to form a second rendering image, and the first rendering image and the second rendering image may be composited so that virtual objects existing in the foreground region are given priority over the real world image and the real world image is given priority over virtual objects existing in the background region. Furthermore, if we assume that there is no error in the marker recognition accuracy and no error in the attachment accuracy of the outer imaging unit (left) and the outer imaging unit (right) to the game device, the position of the outer imaging unit (right) indicated by the right transformation matrix, which is the marker recognition result of the right real world image, will be a position shifted a certain distance (distance between the imaging units) along the x-axis direction of the outer imaging unit (left) coordinate system (the lateral direction of the game device, the horizontal direction when in use) from the position of the outer imaging unit (left) indicated by the left transformation matrix, which is the marker recognition result of the left real world image, and the orientation of the outer imaging unit (right) indicated by the right transformation matrix and the orientation of the outer imaging unit (left) indicated by the left transformation matrix will be the same as the attachment state of the outer imaging unit (left) and the outer imaging unit (right) on the game device. ARActive is a variable (flag) indicating whether or not a virtual object can be compositely displayed on a real world image. In a situation where a virtual object can be compositely displayed on a real world image, the value is set to true, and in a situation where a virtual object cannot be compositely displayed on a real world image (for example, a situation where a marker cannot be recognized at all), the value is set to false (initial value). For example, after calculating the distance between the outer imaging unit (left) and the outer imaging unit (right) in the virtual camera interval determination process, the positions of the left and right virtual cameras may be determined by shifting them in opposite directions by EyeWidth/2 from positions in the virtual space corresponding to the average position of the position of the outer imaging unit (left) calculated based on the marker recognition result for the left real world image and the position of the outer imaging unit (right) calculated based on the marker recognition result for the right real world image.

Similarly, the hardware for the video playback apparatus may be a conventional computing device operating under suitable software instructions, such as a playback processor (e.g., a CPU operating under suitable software instructions) configured to acquire a first video recording of a video game play session including a sequence of video images, configured (e.g., by suitable software instructions) to acquire a second video recording generated by encoding a sequence of depth buffer values, and configured to acquire a sequence of in-game virtual camera positions associated with at least one of the acquired video recordings, and configured to provide the in-game virtual camera positions to an event analyzer and to generate statistical the analysis processor (e.g., a CPU operating under suitable software instructions) configured (e.g., by suitable software instructions) to obtain data representative of statistically significant in-game events and in-game event locations; a position calculation processor (e.g., a CPU operating under suitable software instructions) configured to calculate a location in a current video image of the first video recording that corresponds to the in-game event location responsive to an associated in-game virtual camera position and a decoded depth buffer value; and an augmentation processor (e.g., a CPU and/or GPU operating under suitable software instructions) configured to augment the current video image with a graphical representation of the statistically significant in-game event responsive to the calculated location. Thus, the mapping method may include generating at least a portion of a first map including a graphical representation of some or all of the mapping points of at least a first map data set for display with a first video recording; generating at least a portion of a second map including a graphical representation of some or all of the mapping points of at least a second map data set associated with a different video recording of a different sequence of video images output by the same video game title and sharing the same in-game coordinate system; displaying at least a portion of the first map during playback of the first video recording, the displayed portion including at least a current virtual camera position associated with the displayed video images; and displaying at least a portion of the second map during playback of the first video recording if the respective portion of the second map is within a predetermined range of the current virtual camera position of the in-game coordinate system.

On the other hand, the app allows the user to send a link, for example via email or SMS, to allow a third party cloud to access the real-time mirror-reproduced streaming video. For example, if the user is not identified, a code such as a QR code may be displayed on the monitor so that the user can scan it with a mobile device to download the app. Execution of the app causes the terminal display of the mobile terminal to display application functions (specifically, icons indicating the application functions, etc.) for the mobile terminal to function as a controller. For example, storage may be paused when there is a notification from the user terminal system (power loss, notification from another app, incoming call, etc.). The user may be provided with their own app or portal that allows access to progress data, session data, and appointment schedules.

In addition, in this embodiment, as sensor devices with a sensor function, a chest sensor attached to the user's chest, a wrist analyzer attached to the wrist, and a foot sensor attached to the ankle, shoelaces, soles, etc. are shown, but the present invention is not limited to these, and sensor devices may be attached to any part of the body, such as the waist, upper arm, or neck.

In addition, in this embodiment, as sensor devices having a sensor function, a chest sensor attached to the user's chest, a wrist analyzer attached to the wrist, and a foot sensor attached to the ankle, shoelace, sole, etc. are shown, but the present invention is not limited to these, and sensor devices attached to any part such as the waist, upper arm, neck, etc. may be applied. (Device attached to wrist) As shown in the figure, a device (which may be the detection device shown in the figure or a similar device) may be configured to be attached around the user's wrist, arm, ankle, etc. On the other hand, for example, when determining whether to execute an action based on a change in the movement direction of the user's ankle position, the action may be executed at a timing not intended by the user, such as when the user moves his or her foot slightly, which may cause the user to feel uncomfortable. Alternatively or in addition to this, a tracking sensor attached to a position on the user's body other than near the ankle may be used to determine whether to start executing an action and/or whether to complete the executed action. For example, in a game such as soccer that includes the action of kicking an object, the game processing device may determine that it is a dribble if the kicking action is started from an ankle position lower than the boundary.

The bone model includes a plurality of nodes and bones connecting pairs of nodes, and the nodes are arranged at positions corresponding to the chest, neck NK, head, shoulders, arm joints, wrist joints (arm tips) HD, waist, hip joints, knees, feet FT, and other parts of the virtual character. The processor may change the drawing mode of the knee part of the avatar or the auxiliary line associated with the knee part as the knee movement of the trainee's body approaches a direction along the auxiliary line. Note that instead of a protective sheet, the player may be required to wear a helmet or protective pads for the shoulders, elbows, knees, and the like. The knee and ankle sensors are similarly identified by hemisphere (left or right) and height. Motion sensors may also be attached to both elbows and both knees of the user.

The objects referred to here include, but are not limited to, objects such as a batter, pitcher, catcher, fielder, umpire, ball, mound, home base, first base, second base, third base, fence, and spectator stands that are necessary to generate data of the virtual space in the game. The virtual space data also includes imaging target information, which is various information related to the imaging target, such as the content and schedule of the imaged event, information about the performers and speakers, and information indicating the imaging location. The virtual space data includes data related to each virtual object that constitutes the virtual space, data related to a light source that illuminates the virtual space, and the like. When generating the virtual space data, the processing unit may obtain images of various objects from the storage unit. The virtual space data includes data of a panoramic image captured by an omnidirectional camera.

The user's own team consisting of players owned by the user and the cooperation team are defined by a deck, the deck has a predetermined number of frames for each role constituting the team, and the role of the player is set by linking the player to the frame in the deck, and in the generating step, players may be selected from each user's deck so that the same role as the role set in the user's deck is set for the player in the cooperation team's deck, and players are linked to all the frames of the roles constituting the cooperation team. For example, the string-like member has a structure that prevents the HMD from touching the floor surface of the structure when the HMD is attached or detached from the player, has a length that does not touch the floor surface of the structure even when the HMD is attached or detached from the player, the string-like member is in the shape of an expandable spiral, or has a structure that winds up the cable to adjust the length of the string-like member. As another example, the management unit may, for example, lower the movement performance of each character image linked to each user belonging to the team as the team acquires more points, reduce the size of each tracking character image linked to each tracking player belonging to the tracking team, increase the size of each escaping character image linked to each escaping player belonging to the escaping team, or lower the operability of each character image linked to each user belonging to the team. As another example, when multiple character images linked to multiple users move to a specific location in the virtual space, or multiple users move to a specific location in the real space, and at the specific location, a character image linked to a user is overlapped or connected to a character image linked to another user, points may be awarded to the user or the other user. For example, the management unit may lower the movement performance of each character image linked to each user belonging to a team with a larger number of users, may reduce the size of each pursuit character image linked to each pursuit player belonging to the pursuit team, may increase the size of each escape character image linked to each escape player belonging to the escape team, or may lower the operability of each character image linked to each user belonging to the team.

For the purpose of recreating the experience of enjoying the game of this embodiment together, it is appropriate to reproduce the actual video game as it actually exists, but if the video game is to be realized by a game execution unit installed in the server, it is necessary to create a new program that reproduces the video game, or to create an emulator for a console game in order to execute the program for the video game on the game execution unit installed in the server.

In the figure, each element described as a functional block performing various processes can be configured in hardware with a circuit block, memory, or other LSI, and in software with a system software or a game program loaded in memory. The virtual space image displayed on the display unit of the talent terminal may include a plurality of virtual space images corresponding to the view from various positions (virtual viewpoints) in the virtual space corresponding to a certain handshake event venue. Those skilled in the art will understand that in the various embodiments described in this disclosure, the aforementioned handheld device is utilized in conjunction with an interactive application displayed on the display to provide various interactive functions. In the figure, each element described as a functional block performing various processes can be configured in hardware with a CPU, main memory, or other LSI (Large Scale Integration) stored in the housing of the stereoscopic image presentation device. It should be noted that although some of the above-mentioned embodiments relate to transmitting a signal from a pad device to an HMD, in various embodiments, the signal is transmitted from the pad device to a game machine.

The premium/discount coefficient is a coefficient for moving the hand of the virtual experiencer, which is the body, in the direction of a certain area (e.g., a danger area) more or less than the virtual experiencer intends when the virtual experiencer moves the hand, which is the body, in the virtual space. When the inner product of a unit vector facing in the same direction as the movement amount vector Y(n-1) and a unit vector in the direction of looking at a certain area from the body in the virtual space is greater than a predetermined value, the premium/discount coefficient can be set to a value greater than 1 to move the hand more than the virtual experiencer intends, and the premium/discount coefficient can be set to a value less than 1 to move the hand less than the virtual experiencer intends. In the above embodiment and modified example, before recognition, the marker is displayed on a monitor, the marker displayed on the monitor is imaged and saved, and when recognition is performed, the imaged and saved marker is compared with a newly imaged marker. However, by storing the marker in advance, such a saving process can be omitted, and when recognition is performed, the previously stored marker can be compared with the newly imaged marker. In the above embodiment, the terrain object itself is deformed in the step shown in the figure (i.e., each vertex of the terrain object is moved), so that the moving area is displayed as if it is moving, but in other embodiments, the terrain object itself is not deformed, and instead the moving area may be displayed as if it is moving by changing the texture coordinates used in texture mapping on the terrain object. If the game device has a setting in which the viewpoint is a first-person viewpoint during imaging and a third-person viewpoint during normal times, it becomes easy to view and operate the player object by placing the left and right virtual cameras in positions other than the viewpoint of the player object during normal times, and when saving a stereoscopic image, it is possible to save a stereoscopic image with a sense of realism as if it were seen from the viewpoint of the player object by placing the left and right virtual cameras in positions corresponding to the viewpoint of the player object. When performing the virtual/reality synthesis process of playing an actual musical instrument in a virtual space as described above, if the trained semantic segmentation model unit constantly performs semantic segmentation inference processing on the dual camera image data obtained from the dual cameras attached to the HMD, the inference processing is performed even when the performer is not looking at the instrument or his/her own body, and if an error occurs in the inference result, noise may be mixed into the virtual/reality synthesis image data.

The present embodiment further relates to a computer-implemented method for generating a visual virtual agent configured to perform an artificial physiological color change response to a user for at least one visual virtual agent state and/or for at least one state of the user, the visual virtual agent being configured to perform a visual response to the user, the at least one visual virtual agent state and/or the state of the user being determined based on user-specific parameters including user activity data and/or user physiological and psychological data, the at least one user-specific parameter being acquired using at least one sensor, the visual response being dependent on changes in the user activity data and/or the user physiological and psychological data, and the visual response being dependent on changes in the at least one determined visual and/or a color change of the visual virtual agent for at least one determined state of the user, the method including acquiring user activity data and/or user psychophysiological data over time using at least one sensor, acquiring the user activity data and/or the user psychophysiological data from the at least one sensor with a processor, analyzing the user activity data and/or the user psychophysiological data with a processor, determining a visual virtual agent state based on the received acquired activity data of the user and/or the user psychophysiological data of the user with a processor, and/or determining a visual virtual agent state based on the received acquired activity data of the user and/or the user psychophysiological data of the user with a processor. determining, by the processor, a current state of the user; selecting, by the processor, at least one sensor for generating a color change response based on the user determined current state and/or based on the determined visual virtual agent state; rendering, by the graphic processing unit, the visual virtual agent in a first color for the determined visual virtual agent state and/or the user determined current state; displaying, on the display device, on at least one display, the visual virtual agent in the first color for the determined visual virtual agent state and/or the user determined current state; The method includes the steps of acquiring (in real time) the user's acquired activity data and/or the user's acquired physiological and psychological data by the processor, analyzing (in real time) the user's acquired activity data and/or the user's acquired physiological data from at least one selected sensor by the processor, determining a change in the user's acquired activity data and/or the user's acquired physiological data by the processor, depicting the visual virtual agent in a second color by the graphic processing unit in response to the determined change in the user's acquired activity data and/or the user's acquired physiological data, and displaying on the display device the visual virtual agent in the second color device for the determined visual virtual agent state and/or the user's determined current state.

In addition, in this embodiment, as sensor devices having a sensor function, a chest sensor attached to the user's chest, a wrist analyzer attached to the wrist, and a foot sensor attached to the ankle, shoelaces, soles, etc. are shown, but the present invention is not limited to these, and sensor devices attached to any part of the body, such as the waist, upper arm, or neck, may be applied. In addition, in the case where the tracked character image is an image representing a character originally wearing shoes, when an item (shoes) is acquired, the image of the original shoe part of the tracked character image may change. In addition, in the "kick mode", the movement of the upper body may be reflected in the action, such as by increasing the movement distance of the shoe object according to the movement of the upper body. For example, if the lower leg is long, the movement distance of the shoe object may be corrected to be longer than if the lower leg is short. Conversely, if the lower leg is long, the movement distance of the shoe object may be corrected to be shorter than if the lower leg is short.

In this embodiment, the image device stores a three-dimensional image representing a three-dimensional object in a virtual space as an original image in an internal storage device in advance, and generates a display image from the three-dimensional image by processing to make the image look realistic and natural to the user so that the user has the illusion that the three-dimensional object in the virtual space exists there. However, as a modified example, the display image may be a three-dimensional image consisting of two images with parallax between the left and right eyes. As a result, even though the outer surface of the upper housing faces the front of the player, the game image displayed there is an image taken from above the virtual space, that is, an image that should be displayed when the outer surface of the upper housing is facing downward. Specifically, when driving a cart in a very narrow and poorly visible passage such as a warehouse, a head-mounted display worn by the driver may display a plan view of the passage from directly above, as well as the driver's own cart and other pedestrians and carts moving there. The virtual space construction unit may then estimate what the subject is based on the position and arrangement of the subject part detected from the distance image, and arrange voxels representing the subject based on the estimation result. The vergence and divergence of the two eyes relative to one another (i.e., the rolling of the pupils towards or away from one another to converge the gaze of the eyes and fixate them on an object) is closely linked to the focusing (or "accommodation") of the eye's lenses.

Modifications In the above-described embodiment, an example was given of a game in which a target object can be destroyed when an operation is performed on a first controller provided on an operation unit of a first game device in a state in which the target object and the first mark overlap (an example of a case in which the positional relationship between the display object and the first designated position satisfies the third condition), or when an operation is performed on a second controller provided on an operation unit of a second game device in a state in which the target object and the second mark overlap (an example of a case in which the positional relationship between the display object and the second designated position satisfies the third condition). A game may be played in which, when an operation is continued with a first operator provided on an operation unit of a first game device while the first object and the second mark are overlapping (an example of a case where the positional relationship between the displayed object and the second designated position satisfies the third condition), or when an operation is continued with a second operator provided on an operation unit of a second game device while the first object and the second mark are overlapping (an example of a case where the positional relationship between the displayed object and the second designated position satisfies the third condition), the first object can be moved to follow the movement of the first mark or the second mark, or a moving target object can be stopped. The game execution means (e.g., the game execution unit in the figure) includes a first tracking means (e.g., the head tracking unit in the figure) that tracks the position of the head based on position information of the player's head obtained from a first sensor (e.g., the HMD in the figure) attached to the player's head, a second tracking means (e.g., the arm tracking unit in the figure) that tracks the position of the hand based on position information of the player's hand obtained from a second sensor (e.g., the arm sensor in the figure) attached to the player's arm, a synchronization means (e.g., the synchronization unit in the figure) that synchronizes the first tracking information obtained by tracking the head position by the first tracking means with the second tracking information obtained by tracking the hand position by the second arm tracking means to obtain position information of the head and hand at the same timing, and a game control means (e.g., the game control unit in the figure) that performs control to display images of objects that match the positions of the head and hands obtained by the synchronization means in a virtual space within the game.

The game program according to this embodiment executes, in order to realize a game simulating a competition (soccer game), a placement step in which a computer including an imaging device (camera) and a display device places a player character in a virtual space in which a positional relationship with real space is defined and in which a virtual camera is placed at a position in the virtual space corresponding to the position of the imaging device (camera) in the real space, a modification step in which the angle of view of the virtual camera is changed according to the movement of the imaging device (camera) in the real space, and a display step in which a virtual image of the virtual space captured by the virtual camera is superimposed on a real image of the real space captured by the imaging device and displayed on the display device, and in the modification step, the angle of view of the virtual camera is narrowed as the virtual camera is closer to the area in which the player character is placed.

For example, when the visual virtual agent determines that the user is running, the user's activity data indicates that the user is running, and the activity data can be acquired using a sensor such as a GPS tracker or step sensor, and the visual virtual agent may be further configured to acquire user-specific data, for example from the GPS tracker or step sensor, and thus may be configured to monitor and automatically determine whether the user has stopped running. In this embodiment, under normal circumstances, each enemy is controlled to operate independently, and during a specific event (for example, a scene in which one of the enemies launches a specific attack), the enemy character control means causes the enemy character to launch an attack that changes the viewpoint position of the player character, and moves enemy characters that are within a specific range from the viewpoint position when changed by the attack into the field of view of the player character (=virtual camera) as viewed from the viewpoint position.

Furthermore, in "flight mode", when the left foot position and the right foot position exceed the front boundary and the position of the tracking sensor attached to the head or the tracking sensor attached to the upper body other than the head moves to a position backward from the initial position when the game was started, that is, when the head or the upper body other than the head is tilted in the opposite direction to the direction in which the legs are extended, the speed and distance of movement of the seat object and the virtual person may be increased to reflect the action. Here, the camera control data is defined in a time series from the start point to the end point as shown in the figure, and if the start point data is defined by time and the intermediate point data is defined by time, by setting the time position as the playback position, it is possible to control the movement of the virtual camera along the route from the intermediate point to the end point. When a virtual space including such an occluding object is captured by the virtual camera, the central part is displayed as an enlarged image of a part of the virtual space displayed under normal circumstances, while the peripheral part is occluded by the occluding object and the part of the virtual space displayed under normal circumstances is no longer displayed (or becomes difficult for the user to see).

In this embodiment, a mechanism is described in which when an HMD captures an image of a marker installed in a model room, a three-dimensional model corresponding to the marker is displayed; however, a mechanism may also be used in which an action is detected and a three-dimensional model corresponding to the action is displayed. Furthermore, a mechanism in which the more a user contributes, the more reward he or she receives is easy to understand for users, is fair in terms of the correlation between contribution and reward, and can prevent users from becoming dissatisfied within a group. Details of the mechanism of MR technology are as described in the prior art described above. The mechanism of MR technology is as described in the prior art documents and prior art described above.

The wearer and/or caregiver may also be presented with the opportunity to personalize the user interface of the system as a reward system upon completion of several sessions of the various games offered. In this way, giving more opportunities to skilled users to operate the game can lead to advantageous progress in cooperative matches in cooperative teams, and is more likely to gain understanding and acceptance from each user. Each seeking opportunity can be tracked by storing its location and the time it takes to successfully look at it. In one embodiment, the opportunity can be a match in which two or more users compete.

Modifications In the above embodiment, an example was given in which the second position determination unit determines whether the first line or the second line intersects with a hit determination volume having a relatively narrow third range set on the target object when the normal display control process is performed, and determines whether the first line or the second line intersects with a hit determination volume having a relatively wide fourth range set on the target object when the special display control process is performed. However, the range of the hit determination volume set on the target object may not be changed between the normal display control process and the special display control process, and the range of hit determination set on the first line and the second line may be changed. In addition, the object generated in the broadcaster terminal may be generated in a virtual space in the broadcaster terminal, and information related to the generated object (for example, information normally required to display an object in a virtual space, such as the object's position, size, shape, polygon, etc.) may be sent to a server to generate the object in the virtual space in the server and move the position of the gift, or information related to the generated object may be sent to a viewer terminal to generate the object in the virtual space in the viewer terminal and move the position of the gift. In addition, in a state in which the target virtual object is determined to be pinched at multiple positions (usually, at one position, it is determined to be pinched between multiple fingers of the left hand or FL5, and at the other position, it is determined to be pinched between multiple fingers of the right hand or FR5), if the distance between any of the fingers determined to be virtually pinching the target virtual object at position and any of the fingers determined to be virtually pinching the target virtual object at position varies, the size of the target virtual object in the virtual space may be changed according to the variation in distance.

Variations In addition, in the above-described embodiment, an example was given in which the special position calculation unit determines a special position based on the first designated position and the second designated position, and the display control unit controls an image based on the special position when a special display control process is performed. However, the display control unit may control the image based on the first designated position independently of the second designated position as a normal display control process when the positional relationship between the first designated position and the second designated position is outside a given range, and control the image based on the second designated position independently of the first designated position, and may control the image based on at least one of the first designated position and the second designated position as a special display control process when the positional relationship between the first designated position and the second designated position is within a given range. For example, when normal display control processing is performed, it may be determined whether a relatively narrow third range area with the first straight line as its center line or a relatively narrow third range area with the second straight line as its center line intersects with a hit determination volume having a predetermined range set on the target object, and when special display control processing is performed, it may be determined whether a relatively wide fourth range area with the first straight line as its center line or a relatively wide fourth range area with the second straight line as its center line intersects with a hit determination volume having a predetermined range set on the target object.

Thus, for example, hardware of an entertainment device operable to implement the mapping techniques described herein may be a computing device operating in accordance with suitable software instructions, and may include a video data processor (such as a CPU and/or GPU) configured (e.g., by suitable software instructions) to obtain a first sequence of video images output by a video game title; a camera position data processor (such as a CPU and/or GPU) configured to obtain a corresponding sequence of in-game virtual camera positions from which the video images were created; a depth data processor (such as a CPU and/or GPU) configured to obtain a corresponding sequence of depth buffer values for a depth buffer used by the video game during creation of the video images; and a mapping data processor (such as a CPU and/or GPU) configured (e.g., by suitable software instructions) to obtain, for each of a plurality of videos and corresponding depth buffer values of the obtained sequence, a mapping point corresponding to a selected predetermined set of depth values corresponding to a set of predetermined positions within the respective video images, where for each pair of depth value and video image position, the mapping point has a distance from the virtual camera position based on the depth value, and a position based on the relative position of the virtual camera and the respective video image position, thereby obtaining a map dataset of mapping points corresponding to the first sequence of video images. Thus, the mapping method may include generating at least a portion of a first map including a graphical representation of some or all of the mapping points of at least a first map data set for display with a first video recording; generating at least a portion of a second map including a graphical representation of some or all of the mapping points of at least a second map data set associated with a different video recording of a different sequence of video images output by the same video game title and sharing the same in-game coordinate system; displaying at least a portion of the first map during playback of the first video recording, the displayed portion including at least a current virtual camera position associated with the displayed video images; and displaying at least a portion of the second map during playback of the first video recording if the respective portion of the second map is within a predetermined range of the current virtual camera position of the in-game coordinate system.

Thus, it will be appreciated that since the map points represent the endpoints of lines with links corresponding to depth information on the image, and directions corresponding to the relative positions of the sample pixels on the screen and the optical axis of the virtual camera, as discussed above, whereas the map points represent lines emanating from the position of the virtual camera, each of which represents a distinct line of sight between the camera and the endpoints, color information for points along these lines is available, and color information for corresponding positions on any display surface below these lines can be sampled for each corresponding position in the map.

Therefore, movement of the marker at normal speed within the imaging range of the outer imaging unit can basically be tracked by updating the marker coordinate system and relative relationship information.

In addition, when, for example, a first finger determined to be facing ventrally and a second finger determined to be facing dorsally are both in contact with the target virtual object, and the distance between the coordinate value (x1, y1, z1) of the fingertip in the XYZ coordinate system of the virtual space of the fingertip and the coordinate value (x2, y2, z2) of the fingertip in the XYZ coordinate system of the virtual space of the fingertip is below a predetermined threshold value (at this time, the part of the target virtual object corresponding to the finger facing dorsally is deformed to be located behind the finger, but the position comparison in the Z-axis direction with the finger facing ventrally is performed using the coordinate value before deformation, so the finger is in a semi-transparent composite and drawn state), the information processing device may determine that the target virtual object is virtually pinched between these fingers at the position of the finger (a predetermined range including (x1, y1) and (x2, y2), for example the range of the smallest circle including these), and execute a predetermined process. Thus, the mapping method may include generating at least a portion of a first map including a graphical representation of some or all of the mapping points of at least a first map data set for display with a first video recording; generating at least a portion of a second map including a graphical representation of some or all of the mapping points of at least a second map data set associated with a different video recording of a different sequence of video images output by the same video game title and sharing the same in-game coordinate system; displaying at least a portion of the first map during playback of the first video recording, the displayed portion including at least a current virtual camera position associated with the displayed video images; and displaying at least a portion of the second map during playback of the first video recording if the respective portion of the second map is within a predetermined range of the current virtual camera position of the in-game coordinate system.

In the embodiment described above, when the first game space image displayed on the head mounted display is moved, the transparency of the first game space image is set so that it gradually becomes more transparent from the center of the first game space image toward the ends, thereby making the boundary between the part where the first game space image is displayed and the part where the first game space image is not displayed less noticeable. However, for example, it is also possible to display a different image (such as an abstract pattern or a pattern resembling countless balls and bats) in the part where the first game space image is not displayed, to give the impression that they are approaching.

Similarly, the hardware for the video playback apparatus may be a conventional computing device operating under suitable software instructions, such as a playback processor (e.g., a CPU operating under suitable software instructions) configured to acquire a first video recording of a video game play session including a sequence of video images, configured (e.g., by suitable software instructions) to acquire a second video recording generated by encoding a sequence of depth buffer values, and configured to acquire a sequence of in-game virtual camera positions associated with at least one of the acquired video recordings, and configured to provide the in-game virtual camera positions to an event analyzer and to generate statistical the analysis processor (e.g., a CPU operating under suitable software instructions) configured (e.g., by suitable software instructions) to obtain data representative of statistically significant in-game events and in-game event locations; a position calculation processor (e.g., a CPU operating under suitable software instructions) configured to calculate a location in a current video image of the first video recording that corresponds to the in-game event location responsive to an associated in-game virtual camera position and a decoded depth buffer value; and an augmentation processor (e.g., a CPU and/or GPU operating under suitable software instructions) configured to augment the current video image with a graphical representation of the statistically significant in-game event responsive to the calculated location. Finally, similarly, the event analyzer may be a server or a conventional computing device operating under suitable software instructions, including a receiver (e.g., an Ethernet port operating in cooperation with a CPU under suitable software instructions), an analysis processor (e.g., a CPU operating under suitable software instructions), a selection processor (e.g., a CPU operating under suitable software instructions), and a transmitter (e.g., an Ethernet port operating in cooperation with the CPU under suitable software instructions), wherein the receiver transmits the video game identifiers and one or more associated in-game events and their respective in-game locations from a plurality of video storage devices, respectively. The video game device is configured to receive the video game identifier and the analysis processor is configured to perform analysis on one or more aspects of the in-game events and their respective in-game locations associated with the video game identifier to identify statistically significant in-game events, the receiver is configured to subsequently receive the video game identifier and at least one of the in-game virtual camera location and the in-game player location from the video playback device, the selection processor is configured to select one or more identified statistically significant in-game events associated with the received video game identifier and having an in-game location within a predetermined distance of the at least one received location, and the transmitter is configured to transmit data representative of the in-game events and their in-game locations to the video playback device.

In one example of this embodiment, it is not necessary for the character posture setting unit to use the position of the user's body part in the real space and the scaling coefficient to determine the position coordinates of all of the above-mentioned nodes of the virtual character in the virtual space. Instead, the position of a preselected selected node among the nodes included in the bone model of the virtual character may be determined in the virtual space using the estimated position of the user's body part in the real space and the scaling coefficient, and then the position of each node included in the bone model of the virtual character other than the selected node may be determined based on the position of the selected node determined in the virtual space. In addition, the object generated in the broadcaster terminal may be generated in the virtual space of the broadcaster terminal, or information related to the generated object (for example, information normally required to display an object in a virtual space, such as the object's position, size, shape, polygon, etc.) may be sent to the server, where the object is generated in the virtual space of the server and the position of the gift is moved, or information related to the generated object may be sent to the viewer terminal, where the object is generated in the virtual space of the viewer terminal and the position of the gift is moved.

According to this aspect, when a user wearing a head-mounted display needs to keep watching a specific object in a virtual space, such as an opponent in a fighting game or a racing car in a racing game, the user wearing the head-mounted display can control the attitude of the head-mounted display so that a virtual point is set at a position corresponding to the specific object in the virtual space, thereby making it possible to continuously display an area in the virtual space where the specific object exists on the display unit of the head-mounted display. As described above, the object generated in the server device may be generated in the server device and the gift may be moved, or information related to the generated object (for example, information normally required to display an object in a virtual space, such as the object's position, size, shape, polygon, etc.) may be sent to a viewer terminal or a distributor terminal, and the object may be generated in the virtual space in the viewer terminal or distributor terminal, and the position of the gift may be moved. For example, all of the modules of the virtual camera control system may be included in a user terminal device (e.g., a desktop computer, a notebook computer, a workstation, a PDA, a web pad (particularly a smart pad), a mobile phone (particularly a smartphone), etc.) (not shown) that typically requires the utilization and control of a virtual camera, but some of the modules may be included in other digital devices (e.g., a three-dimensional content providing server) (not shown) other than the user terminal device that can communicate with the user terminal device.

In this embodiment, the rendering camera setting unit obtains the rendering camera position and the information representing the direction of the field of view (e.g., vector information starting from the rendering camera position and passing through the center of the field of view) that are predetermined (e.g., may be hard-coded in the program or may be read from what is written in the setting file) regardless of the position of the image sensor included in the camera, and sets these as the field of view information. Furthermore, the "device" may be in the form of specially designed hardware that is a general-purpose hardware specialized for executing software that enables the hardware to perform a specific function, software stored on a computer-readable medium, or a combination thereof. Output from the AI system may be passed to a "hardwired" system (here, as part of a behavior processing engine), which can implement a set of rules and output a final social cue. The code module or any type of data may be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device, including a hard drive, solid-state memory, random access memory, read-only memory, optical disk, volatile or non-volatile storage device, combinations of the same, and/or the like.

Here, computers include computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

It is also described that when a player points the controller at the virtual game space (pseudo-dimensional image) with the aiming cursor and, while pressing the A button, swings the controller up and then down, a second beam different from the first beam is shot toward the position in the virtual game space pointed to by the aiming cursor (or a weapon such as a chain with a chain attached to the end may be fired). When an online game realized by the game system is one in which a user operates a character selected from a plurality of characters and plays against other characters, the game progress information 42d may include at least one of character use information, character training progress information, and character setting information (appearance such as costume, weapon, hairstyle, facial parts, character name, skills, etc.). Here, the first state information includes at least one of the following: the physical strength or durability value of the first class object (if the first class object is a virtual person, the physical strength value of the first class object is displayed, and if the first class object is a virtual object such as a machine, weapon, vehicle, etc., the durability value of the first class object is displayed), the number of virtual resources of the first class object, the defense value of the first class object, and the tool information of the first class object. Here, "action" includes moving each part of the body, changing posture, changing facial expression, moving, speaking, touching or moving objects arranged in the virtual space, using weapons, tools, etc. held by the character, and the like. In addition, when a right-handed player plays a game with a weapon in his right hand, a screen configuration in which the character is displayed on the left side of the screen and other objects (map, enemies) are displayed on the right side of the screen as shown in the figure is considered to make gameplay easier for the right-handed player.

The system will preferably be configured to recognize that the user is holding an item and to understand the type of interaction the user is having with the item. Furthermore, in the case of a setting that does not allow items to be dropped (an example of a performance in which the user jumps high and jumps onto something), the information processing device determines that the user's body is rising in real space if the difference between the real reference position information and the real reference position information Pp at the time of the previous execution, which is repeatedly executed at a predetermined timing, is P-Pp>0, and executes the same processing as already described. In addition, since it may be difficult to operate if the inertial coordinate system moves during operation, the on/off of this control mode may be controlled so that the user performs an input operation using, for example, an input controller, or it may be controlled to be off when an operation is being performed on a virtual object. Alternatively, when the generation unit is transmitted movement information indicating that the user has made a movement such as kicking something, it generates image information (video) showing the user kicking an object that exists on the trajectory of the user's foot in the virtual reality space or the augmented reality space. In this embodiment, the system is described as displaying a 3D model corresponding to a marker placed in a model room when the HMD captures the image of the marker, but it may also be a system that detects an action and displays a 3D model corresponding to that action.

The method of the server determining whether or not a second object exists between the target virtual object and the target virtual resource includes, but is not limited to, acquiring a connecting line between the target virtual object and the target virtual resource based on the first reference coordinates and the second reference coordinates, determining whether or not a second object exists on the connecting line, and determining that a second object does not exist between the target virtual object and the target virtual resource if the second object does not exist on the connecting line. Here, the target pace extracted from the target pace database may be generated based on the latest running distance and running time of the user's past running training data, or, if the target pace database stores the topographical information (elevation, course shape, etc.) of the running course and the weather information (temperature, wind direction, wind speed, etc.) associated with each other as the training data, the target pace may be generated based on the running distance and running time extracted based on various conditions such as the topographical information and weather information input by the user.

Then, in step 3, when the running speed reaches the target value, the CPU causes the image generating unit to enlarge the display size to a preset standard size so that the virtual runner appears to be running directly in front of the user's field of vision, and generates a moving image (standard image) of the virtual runner synchronized with the user's motion state by matching the timing of the virtual runner's feet to the timing of the user's feet and matching the pitch of the virtual runner to the user's pitch based on the user's pitch and footsteps. Here, the display command is for generating first visual information from the data of the first type object included in the display command, generating second visual information from the data of the second type object included in the display command, and instructing the terminal to display the first visual information on the user interface or to display the first visual information and the second visual information when the first type object is not present within the field of vision of the visual angle corresponding to the target virtual object, and the amount of information of the first visual information is greater than the amount of information of the second visual information.

Then, in a step where the running speed reaches the target value, the CPU uses the image generation unit to generate a moving image of the virtual runner with an enlarged display size so that the virtual runner appears to be running directly in front of the user's field of vision, and based on the user's pitch and foot take-off and landing timings, matches the virtual runner's foot take-off and landing timings to the user's foot take-off and landing timings, and matches the virtual runner's pitch to the user's current pitch, to generate a moving image of the virtual runner synchronized with the user's motion state.

For example, in the case of a virtual space simulating a baseball field, one of the "first virtual space image" and the "second virtual space image" may be an image showing the virtual space as seen from the position of one of the player characters (e.g., a catcher character or a batter character, etc.) or an umpire character (e.g., a head umpire character), and the other may be an image showing the virtual space as seen from another position (e.g., the position of the stands). In the case of a full-scale baseball game rather than a baseball board game, if the first user wants to curve the virtual object thrown by the first player character to the right, the left controller is used to input a direction to the right (e.g., an input to tilt the analog stick to the right), which causes the virtual object to move on a trajectory that curves to the left as seen by the right-handed first player character (curve ball, slider ball). Generally speaking, the method for the server to execute the game program of this baseball game includes the steps of grouping multiple users who cooperate in a cooperative match based on at least one of the play history in the competitive game of each user and the players owned by each user, generating a cooperative team including one or more players owned by each of the multiple grouped users, and progressing a match between the cooperative team and the opposing team. Specifically, for example, in the case of a virtual space simulating a baseball field, an image showing the virtual space in a state where the first team is playing defense may be the "first virtual space image," and an image showing the virtual space in a state where the second team is playing defense may be the "second virtual space image."

In other words, if the input device is lowered, such as in a golf swing stance, the candidate surface calculated using the Y' axis may be more relied upon to calculate a new reference plane, whereas if the input device is raised, such as in a baseball batter's stance, the candidate surface calculated using the Z axis may be more relied upon to calculate a new reference plane.

Modifications In the above embodiment, an example was given in which the second position determination unit determines whether the first line or the second line intersects with a hit determination volume having a relatively narrow third range set on the target object when normal display control processing is performed, and determines whether the first line or the second line intersects with a hit determination volume having a relatively wide fourth range set on the target object when special display control processing is performed. However, the range of the hit determination volume set on the target object may be left unchanged between normal display control processing and special display control processing, and the range for hit determination set on the first line and the second line may be changed.

Variations In addition, in the above-described embodiment, an example was given in which the special position calculation unit determines a special position based on the first designated position and the second designated position, and the display control unit controls an image based on the special position when a special display control process is performed. However, the display control unit may control the image based on the first designated position independently of the second designated position as a normal display control process when the positional relationship between the first designated position and the second designated position is outside a given range, and control the image based on the second designated position independently of the first designated position, and may control the image based on at least one of the first designated position and the second designated position as a special display control process when the positional relationship between the first designated position and the second designated position is within a given range. For example, when normal display control processing is performed, it may be determined whether a relatively narrow third range area with the first straight line as its center line or a relatively narrow third range area with the second straight line as its center line intersects with a hit determination volume having a predetermined range set on the target object, and when special display control processing is performed, it may be determined whether a relatively wide fourth range area with the first straight line as its center line or a relatively wide fourth range area with the second straight line as its center line intersects with a hit determination volume having a predetermined range set on the target object.

Modifications In the above-described embodiment, an example was given in which the second position determination unit performs a normal position determination process to determine the positional relationship between the target object and the first designated position when the first mark and the second mark are displayed, and to determine the positional relationship between the target object and the second designated position, and performs a special position determination process to determine the positional relationship between the target object and the special position when a special mark is displayed. However, even when a special mark is displayed, the second position determination unit may determine the positional relationship between the target object and the first designated position, and may also determine the positional relationship between the target object and the second designated position.

Modifications In addition, in the above-described embodiment, an example was given in which the update unit updates a given parameter in a first update mode when a normal display control process is performed, and updates a given parameter in a second update mode when a special display control process is performed. However, the display device may further include an attribute setting unit that sets one of a plurality of attributes in association with each of the first and second designated positions, and an update mode change unit that changes the update mode of at least one of the first and second update modes based on a combination of attributes associated with each of the first and second designated positions.

It was very difficult to secure a person who meets the abilities and conditions required for a pacemaker, that is, a runner with the same or higher running ability as the user and who can understand the user's ability and provide appropriate guidance. However, by applying the above-mentioned exercise support device and exercise support method, a pacemaker that is optimal for oneself can be easily used. The biorecovery ability data can include at least one of the following: data that is generally applicable according to the gender, age, and ongoing athletic event of the athlete, and personal data (Personal Data), which is data measured during the existing competition history or training performed by each athlete and accumulated individually. In the generating step, the order is determined in consideration of the attributes linked to each athlete, and the attributes may include at least one of a parameter representing the athlete's ability and a role in the team set for the athlete. This is because, in order to contribute a desired athlete with high ability to the cooperation team, it is more effective to evenly place players with a certain ability or higher in all positions in the deck than to biasedly strengthen a specific player in a specific position. For this reason, when a character belonging to the user team is in possession of the ball, the evaluation value calculation unit may increase the evaluation value of a partial area in which a teammate character whose offensive ability is higher than the standard (i.e., whose offensive ability parameter value is higher than the standard value) is located.

[Process for Referencing the Type of Virtual Object Set for Other Player Attribute Information] Note that in the above example, when there are multiple objects in real space that are expected to be replaced with a virtual object of a specific type (such as an NPC), it is decided at random for each group identifier (player attribute information) which object will be replaced with the virtual object of the specific type and placed thereon. However, this embodiment is not limited to this, and the server device may obtain information such as whether or not there is a group identifier that is currently being replaced with a virtual object of the specific type and placed thereon, or how many such group identifiers there are for each object, and may refer to this information to select an object that is not currently being replaced with a virtual object of the specific type for another group identifier, and may then replace the selected object with a virtual object of the specific type and place it thereon.

[User] In the above examples, the amount of movement in the virtual space is determined based on the amount of movement of an object (which may be a part of the user) that moves with the user, but this embodiment is not limited to this. The information processing device may use the amount of movement of an object that does not move with the user, such as a drone that moves in real space, and may determine the amount of movement of a virtual object in the virtual space corresponding to the object, such as a drone, by multiplying the amount of movement in a predetermined axis direction by a predetermined factor, and may move the virtual object by the determined amount of movement and present an image representing the state in which it has been moved to the user. This virtual environment may be used to simulate an actual environment in reality, and for example, this virtual environment may include the sky, land, or sea, and the land may include environmental elements such as a desert or a city, and the user may control the virtual object to move in this virtual environment, and this virtual object may be a virtual avatar that represents the user in this virtual environment, and this virtual avatar may be any form, such as a human or animal, and this application is not limited to this. In another example of this embodiment, instead of this calculation, when determining position coordinates in virtual space corresponding to the user's focused part at the real reference position in real space, the sign of the relative position from the real reference position to the foot position (information determined by the processing in the figure) is inverted, multiplied by a predetermined ratio, and further converted to the scale of virtual space to obtain information on the relative position to the real reference position as seen from the foot position (value in the world coordinate system of the virtual space), and the information on the virtual reference position determined by the processing in the figure is added to this to determine position coordinates in virtual space corresponding to the user's focused part at the real reference position in real space.

In addition, in the above embodiment, the angle difference (|60 degrees - 30 degrees| = 30 degrees) between the controller pitch angle at which the virtual camera pitch rate is 0 (the shooting direction is horizontal) and the controller pitch angle corresponding to the upper limit of the virtual camera's movable range was the same as the angle difference (|0 degrees - 30 degrees| = 30 degrees) between the controller pitch angle at which the virtual camera pitch rate is 0 and the controller pitch angle corresponding to the lower limit of the virtual camera's movable range, but this embodiment is not limited to this, and the angle difference between the controller pitch angle at which the virtual camera pitch rate is 0 and the controller pitch angle corresponding to the upper limit of the virtual camera's movable range may be larger or smaller than the angle difference between the controller pitch angle at which the virtual camera pitch rate is 0 and the controller pitch angle corresponding to the lower limit of the virtual camera's movable range.

The server may then extract skills that the user lacks relative to the target state based on the skill proficiency and characteristics obtained by the analysis, and may present the trainee with advice (e.g., advice on correcting form, such as "keep your wrists higher," "don't raise your shoulders," "don't move your head back and forth," and "don't round your spine") and practice menus (e.g., practice menus and practice schedules for improving the lacking skills) based on the extracted results. For example, if a heat reduction skill (second attribute information) that can halve the effects of the attribute "high temperature" is associated with the player character, and an equipment object attached to the player character or hand object is associated with a heat reduction equipment effect (third attribute information) that can halve the effects of the attribute "high temperature," the processor may add up the skill and the equipment effect and determine that the effects of the attribute "high temperature" can be nullified. In the case where the online game realized by the game system is one in which the user operates one character selected from among multiple characters and plays against other characters, the game progress information may include at least one of character usage information, character training progress information, and character setting information (appearance such as costume, weapon, hairstyle, facial parts, character name, skills, etc.). Such skill exploration practice can be said to be a process of trial and error exploration of how to use the body, posture, how to direct attention, and how to use the senses (what to listen to, when and where to look, etc.) to reduce unnecessary muscle tension during performance, reduce mistakes, and create the desired sound and expression. For example, if skill A and skill B (for example, skill B and skill B are skills (skills) for playing different melodies) are practiced consecutively with the same amount of practice, the establishment of skill B may hinder the establishment of skill B, and as a result, only skill B may be established.

Thus, advantageously, mapping points or map data sets can be included for at least a portion of the map corresponding to image data not yet displayed in such a way that the playback device can access the mapping points, and optionally altitude data, color data, etc. (i.e., access map information corresponding to the current position shown in the video, but also image/depth/camera information for positions before or after the current position that would otherwise be required). In other implementations, the wearable system can determine context coefficients associated with objects in the user's environment and enable automatic cone updates, in addition to or as an alternative to manual updates, which can advantageously make it easier for the user to interact with objects in the environment, since less user input is required. Note that the reward points are not limited to the above examples and may be experience points for increasing the user's level, virtual currency that can be exchanged for various items in the game, or other in-game value used to advance the game. In this way, when the parameters used in the game calculations are set to values that are more advantageous to the player character in the game than in the normal state, the player character may not be hit by an attack that would normally hit him, or may not receive damage even if it does, or may receive less damage than in the normal state. Special points, etc. are locations in the virtual space that are advantageous in the game, and may be set in positions that are in the shadows of defensive wall objects such as walls, buildings, trees, cars, and rocks, making them less susceptible to enemy attacks.

Here, the camera control data is defined in a time series from the start point to the end point as shown in the figure, and if the start point data is defined by the time and the intermediate points data is defined by the time, the virtual camera can be controlled to move from the intermediate point of the movement route to the end point by setting the time position as the playback position. Also, as an example, if at least one of the object position, object orientation, virtual camera position, and virtual camera orientation changes during the operation to transform the object, parameters for scaling the object may be calculated by changing any of the object's scaling axis, expansion direction, and center position, etc., according to the state of the object and/or virtual camera after the change. As another example, even if at least one of the object position, object orientation, virtual camera position, and virtual camera orientation changes during the operation to transform the object, the process of scaling the object may be continued using the state of the object and/or virtual camera before the change. Also, for example, in the process of sequentially referring to the luminance values of pixels from pixel P(n-8) to the left to find the maximum luminance value (the first maximum), if the luminance value of the reference pixel exceeds a predetermined value (e.g., 250), the process of finding the maximum luminance value may be interrupted and a white area luminance value may be calculated based on the luminance value of the reference pixel. Also, if the grab position goes outside the display range of the display during an operation to transform an object, the process of changing the enlargement or reduction amount may be stopped at the point where the grab position goes outside the display range (i.e., the size immediately before the grab position goes outside the display range is maintained).

From the perspective of efficiently burning fat through aerobic exercise, it is most important to get oxygen into the blood, and care must be taken to avoid anaerobic exercise by overpacing. The HP system may use data from both the FDT and the VIO to increase the accuracy, speed, and/or efficiency of the HP system in determining head posture, angular velocity, and/or linear velocity. For example, a mathematically definable surface may be an efficient (e.g., compared to a large-scale point cloud) and digestible input to other processing devices such as a game engine. For example, a mathematically definable surface may be an efficient (e.g., compared to a large-scale point cloud) and digestible input to other processing devices such as a game engine. However, in order to perform efficient training, it is important to determine the scenes to focus on in accordance with the player's movements.

The synchronization frequency determination unit sets the synchronization frequency to a first frequency when the second user is within a predetermined angle range including the line of sight of the first user in the virtual reality space and closer than a predetermined distance from the first user, sets the synchronization frequency to a second frequency lower than the first frequency when the second user is within the predetermined angle range and farther than the predetermined distance in the virtual reality space, and sets the synchronization frequency to a third frequency lower than the first frequency when the second user is outside the predetermined angle range in the virtual reality space.

The synchronization frequency determination unit sets the synchronization frequency high when the second user is within a predetermined angle range including the line of sight of the first user in the virtual reality space and closer than a predetermined distance from the first user, and sets the synchronization frequency low when the second user is within a predetermined angle range and farther than a predetermined distance in the virtual reality space. That is, the synchronization frequency determination unit sets a first delivery frequency for users and U3 who are in the near-distance area within the field of view, sets a second delivery frequency for users and U5 who are in the medium-distance area within the field of view, sets a third delivery frequency for users to U9 who are in the far-distance area within the field of view, and sets a fourth delivery frequency for users Uj (j≧10) who are in the out-of-field area. Based on the user's answer results, if the user has relatively little exercise experience (the period of time that the user has played sports is less than a certain period, the habit of exercising is less than a predetermined frequency, etc.), the server may suggest to the user training items included in the training item information that have a relatively small load on the body or a relatively low level of difficulty. In the above example, once a background voxel is placed, it is not deleted. However, it is also possible to check for the presence of a subject part corresponding to the background voxel at a lower update frequency than the update process for the person voxel, and delete background voxels that are placed at candidate positions where the presence of a subject part is no longer detected.

The embodiment of the present application mainly relates to an electronic game or an analog training scene, and takes an electronic game scene as an example. The user can perform actions in advance on the electronic device, and the electronic device can download a game setting file for the electronic game after detecting the user's actions. The game setting file may include the application program, interface display data, or virtual environment data of the electronic game, and the user can call up the game setting file and render and display the electronic game interface when logging in to the electronic game on the electronic device. In this embodiment, the main unit performs a correction to reduce the tension in the vertical direction, but in other embodiments, a correction to reduce the tension in any direction according to the game content and game settings may be performed. Note that the position coordinates of the viewpoint and the position coordinates of the virtual camera do not necessarily need to be completely the same, and there may be some difference (for example, a difference of about 10%) between the position coordinates of the two within the range of adjustment of the game settings. Note that the coefficient and the coefficient may be set appropriately according to the game content and game settings, and the coefficient may be set larger or smaller than the coefficient. For example, the game setting means performs processing to request the server device to register a user, and processing to request the server device to delete information that can identify the user.

Here, the game processing unit performs various game processes such as coin (cost) acceptance process, various mode setting process, game progress process, selection screen setting process, process for determining the position and rotation angle (rotation angle around X, Y or Z axis) of an object (one or more primitive surfaces), process for moving an object (motion process), process for determining the viewpoint position (virtual camera position) and line of sight angle (virtual camera rotation angle), process for placing objects such as map objects in object space, hit check process, process for calculating game results (achievements, achievements), process for multiple players to play in a common game space, and game over process, based on operation data from the operation unit, personal data from the save information storage device, game program, etc. In addition, in this case, by registering a panoramic image, the user may be given a privilege (for example, coins that can be used in the game, special items, a predetermined parameter increase, etc.). Furthermore, the amount of coins given to the user is specified, for example, based on the degree of support that the user has given to the player.

More specifically, the image recognition process may be a process of calculating the position of the player's face, eyes, and mouth as a result of the above process, judging the player's facial expression (whether the player is smiling, talking, or blinking), or a process of performing personal authentication (identifying the player as a previously registered person). While viewing the content (panoramic video) in the virtual space, the user first decides the angle at which they want to shoot (the position of the virtual camera), and data is obtained regarding the timing of shooting the panoramic video, such as who wants to shoot from what position and when (or when the video was shot). The angle information (for example, information indicating who shot from what position, at what time, and who shot) is associated with the ID of the content and stored in the server for each content. Alternatively, a real user may send an invitation in the real space, and information regarding who was invited by whom may be added to the user information.

The present embodiment further relates to a computer-implemented method for generating a visual virtual agent configured to perform an artificial physiological color change response to a user for at least one visual virtual agent state and/or for at least one state of the user, the visual virtual agent being configured to perform a visual response to the user, the at least one visual virtual agent state and/or the state of the user being determined based on user-specific parameters including user activity data and/or user physiological and psychological data, the at least one user-specific parameter being acquired using at least one sensor, the visual response being dependent on changes in the user activity data and/or the user physiological and psychological data, and the visual response being dependent on changes in the at least one determined visual and/or a color change of the visual virtual agent for at least one determined state of the user, the method including acquiring user activity data and/or user psychophysiological data over time using at least one sensor, acquiring the user activity data and/or the user psychophysiological data from the at least one sensor with a processor, analyzing the user activity data and/or the user psychophysiological data with a processor, determining a visual virtual agent state based on the received acquired activity data of the user and/or the user psychophysiological data of the user with a processor, and/or determining a visual virtual agent state based on the received acquired activity data of the user and/or the user psychophysiological data of the user with a processor. determining, by the processor, a current state of the user; selecting, by the processor, at least one sensor for generating a color change response based on the user determined current state and/or based on the determined visual virtual agent state; rendering, by the graphic processing unit, the visual virtual agent in a first color for the determined visual virtual agent state and/or the user determined current state; displaying, on the display device, on at least one display, the visual virtual agent in the first color for the determined visual virtual agent state and/or the user determined current state; The method includes steps of acquiring (in real time) the user's acquired activity data and/or the user's acquired physiological and psychological data by the processor, analyzing (in real time) the user's acquired activity data and/or the user's acquired physiological data from at least one selected sensor by the processor, determining a change in the user's acquired activity data and/or the user's acquired physiological data by the processor, depicting the visual virtual agent in a second color by the graphic processing unit in response to the determined change in the user's acquired activity data and/or the user's acquired physiological data, and displaying on the display device the visual virtual agent in the second color device for the determined visual virtual agent state and/or the user's determined current state.

Additionally or alternatively, the wearable device may use distance, size of periocular features, and/or other factors to determine whether the wearable device is in place or removed. The body may have a sponge material that can easily evacuate sweat and heat from the athlete and absorb external shocks, and a portion of the body may be waterproofed to protect the terminal from moisture caused by sweat from the athlete and environmental factors such as rain. These and/or other factors may be used alone or in combination to determine whether movement has occurred, is occurring, or will occur. These exemplary factors may be used alone or in combination to determine the relative position between the HMD and the user's face. These factors may create additional difficulties for individuals as they may not mirror and establishing a connection with other people may be more difficult.
There are various methods of aerobic exercise, such as jogging and swimming, but walking is particularly popular because it is easy to do.

Note that, if possible in terms of the processing capacity of the display control device, a method for updating the first virtual space image displayed on the head mounted display in accordance with the orientation of the user's head may involve, as with the second virtual space image, acquiring a new first virtual space image in the viewing direction corresponding to the changed orientation of the head mounted display, and updating the first virtual space image displayed on the head mounted display to the newly acquired first virtual space image; however, this method requires the first virtual space image to be newly acquired along with the second virtual space image, which increases the processing load. A method for updating the first game space image displayed on the head mounted display in accordance with the orientation of the user's head may involve, as with the new second game space image, acquiring a new first game space image representing the game space in the viewing direction corresponding to the changed orientation of the user's head, and updating the first game space image displayed on the head mounted display to the newly acquired new first game space image; however, this method requires the new first game space image to be newly acquired along with the new second game space image, which increases the processing load. As a method for updating the first game space image displayed on the head mounted display to match the orientation of the user's head, a method can be considered in which, similar to the new second game space image, a new first game space image corresponding to the changed orientation of the head mounted display is newly acquired, and the first game space image displayed on the head mounted display is updated to the newly acquired new first game space image; however, this method increases the processing load because it is necessary to newly acquire a new first game space image along with the new second game space image. Pe(t=n): position of the second user to be corrected this time (current corrected gaze information) Pe(t=n-1): position of the second user corrected last time (previous corrected gaze information) Ps(t=n-1): position of the second user received last time (previous gaze information) Re(t=n): head direction of the second user to be corrected this time (current corrected gaze information) Re(t=n-1): head direction of the second user corrected last time (previous corrected gaze information) Rs(t=n-1): head direction of the second user received last time (previous gaze information) Fs: weighting function: weighting function: weighting function: weighting functionHere, the position of the second user and Ps are expressed in three-dimensional coordinates in virtual reality space. The server may then extract the skills that the user lacks relative to the target state based on the skill proficiency and characteristics obtained through the analysis, and provide the trainee with advice (e.g., advice on correcting form, such as "keep your wrists higher," "don't raise your shoulders," "don't move your head back and forth," and "don't round your spine") or a practice menu (e.g., a practice menu or practice schedule for improving the lacking skills) based on the extracted results.

When resetting the orientation of the virtual camera, only the yaw angle is reset because if the pitch angle and roll angle are also reset, the orientation of the virtual camera in the virtual space cannot be changed to correspond to the change in the orientation of the main device in the real space. When it is not determined in step that the remaining time of the cooldown set for the tapped portal is 0 minutes, or when it is not determined in step that the distant play flag associated with the portal has been reset, the process proceeds to step and the ticket consumption inquiry screen shown in the figure is displayed superimposed on the touch screen. For example, when the selection of the target object is completed in the game progress process (in the case of step), when it is determined that the mission clear condition is established (in the case of step), etc., the elapsed time measured to provide information may be reset. In this embodiment, the orientation of the virtual camera is reset, but the background object in the virtual space may be moved so that the horizontal component of the vector indicating the direction of the line of sight coincides with the reference direction. To reduce deviations, the gyroscope is periodically reset, and the resetting may be performed using other available information such as position/orientation determination based on visual tracking of the object, accelerometer, magnetometer, etc.

That is, for example, if tree polygon models to be placed as fixed objects in a field are calculated from the model position information of the tree placement file to be model numbers 1, 2, 3, 4, 5, etc., based on the placement position data of the virtual camera, when the virtual camera is at position, the tree polygon models that should be selectively acquired as images on the projection plane will be the three tree polygon models (model number and 3); when the virtual camera is at position, the number of tree polygon models that should be acquired as images on the projection plane will be the three tree polygon models (model number and 4); and when the virtual camera is at position, the number of tree polygon models that should be acquired as images on the projection plane will be the three tree polygon models (model number and 5). For example, when the corresponding relative relationship information is not acquired in the current frame, but the position where the marker should be is not included in the captured image, it is possible that the marker is simply not captured in the captured image and has not been removed, so the deletion unit does not delete the marker coordinate system. When placing a deselected component on a background object, the image generation unit may determine whether the placement position specified by the controller object (the position where the component is originally supposed to be placed) matches a prohibited placement position, in other words, whether it is included in an area indicated by a prohibited placement position. In the processes or operations described above, the processes or operations can be freely changed as long as no inconsistencies in the processes or operations occur, such as the use of data that should not yet be available in a certain step. For example, an object showing the center circle that should have been along the circumference of the marker is not along that circumference.

Various Aspects The computer program according to the first aspect can "be executed by at least one processor to cause at least one processor to function as follows: acquire motion information relating to the motion of each of multiple body parts output from multiple first sensors attached to multiple body parts of the broadcaster's body present in physical space; acquire position information relating to the position of the body output from at least one second sensor attached to the broadcaster's body; calculate the position of each of the multiple body parts in physical space based on the motion information; and calculate the position of the body in physical space based on the position information."

The online game image may further include various other objects such as game characters such as NPCs (non-player characters) that appear in the online game, scenery such as buildings and trees, furniture in a room and interior walls, items lying on the ground, parts of the body (hands and feet) of the avatar (the user himself) operated by the user, and objects held by the avatar (guns and swords). The various information managed by the server includes the position, posture, and shape of real objects in a virtual space that is aligned and superimposed on the real space (or a space where the virtual space is aligned and superimposed on the real space = MR space (in other words, a space where the real space and the virtual space are combined)). Note that the points as rewards are not limited to the above examples, and may be experience points for raising the user's level, virtual currency that can be exchanged for various items in the game, or other in-game values used to advance the game advantageously. Alternatively, an information processing device may read markers or QR codes (registered trademark) stored in facilities at various locations, transmit the read data to a search server, and the search server may execute a database search based on the received data and provide access information for obtaining information in the vicinity of the location where the received marker or QR code (registered trademark) is stored.

In the above example, the necessity of controlling the upward movement of the virtual camera in response to the entry into the low-angle area may be determined depending on the type of character (e.g., the type of clothing worn, etc.). Here, the characters are assumed to be, for example, a cursed and imprisoned monster, and an avatar that speaks and moves in response to the actor's voice and actions. For example, the input device may be something that the player holds and moves, or it may be something that the player wears and moves. The game medium may be, for example, an item such as clothing that can be worn by the character. The object may be a simple character icon that represents a talent or a fan with a two-headed figure, or a sophisticated character icon.

REFERENCE SIGNS LIST 10 Virtual space moving device 11 Shoes 12 Scent generating device 13 Character 14 Moving course information 15 Accessory information 100 Storage unit 200 Acquisition unit 300 Calculation unit 400 Control unit 500 Output unit 600 Purchase unit 700 Direction indication unit 800 Connection unit

Claims (6)

A storage unit that stores movement course information including scenery when a character moves in a virtual space in cooperation with a user;
An acquisition unit that acquires biological information indicating a biological state of a user who is exercising and exercise information indicating an amount of exercise of the user;
a calculation unit that calculates a virtual transformation distance in the virtual space based on the motion information;
a control unit that changes a viewpoint in the virtual space by moving a virtual camera in the virtual space from a first position to a second position according to the virtual transformation distance;
an output unit that outputs the movement course information and the character;
the control unit changes the viewpoint and changes a light source setting of movement course information output to the output unit, and includes a direction instructing unit that instructs the user on a movement direction of the user via the output unit based on the movement course information along which the character has moved;
The control unit of the virtual space movement device designates a character based on the bio-information and motion information, and displays a character with a warning meaning when the bio-information becomes a physically abnormal value . 2. The virtual space movement device according to claim 1 , wherein the control unit changes the viewpoint and controls sound effects output to the output unit. the virtual space movement device includes shoes that apply impacts to soles of the user's feet;
3. The virtual space movement device according to claim 2 , wherein the control unit controls the type, magnitude, and timing of the impact of the shoes based on the movement course information. The virtual space movement device according to claim 1, wherein the motion information includes any one of the walking pace, number of steps, and posture obtained when the user actually walks. The computer
a storage step of storing movement course information including scenery when the character moves in the virtual space in cooperation with the user;
An acquisition step of acquiring biological information indicating a biological state of a user who is exercising and exercise information indicating an amount of exercise of the user;
a calculation step of calculating a virtual transformation distance in the virtual space based on the motion information;
a control step of changing a viewpoint in the virtual space by moving a virtual camera in the virtual space from a first position to a second position according to the virtual transformation distance;
an output step of outputting the movement course information and the character;
the control step includes a direction instructing step of changing the viewpoint and changing a light source setting of movement course information outputted in the output step, and instructing the user on a movement direction of the user via the output step based on the movement course information along which the character has moved;
The control step specifies a character based on the bio-information and the motion information, and, if the bio-information becomes a physically abnormal value, displays a character with a warning meaning . On the computer,
A memory function that stores information about a course of movement of a character, including scenery, when the character moves through a virtual space in cooperation with a user;
An acquisition function for acquiring biological information indicating a biological state of a user who is exercising and exercise information indicating an amount of exercise of the user;
A calculation function for calculating a virtual transformation distance in the virtual space based on the motion information;
a control function of changing a viewpoint in the virtual space by moving a virtual camera in the virtual space from a first position to a second position according to the virtual transformation distance;
an output function for outputting the movement course information and the character;
the control function includes a direction instruction function for changing the viewpoint and changing a light source setting of movement course information output to the output function, and instructing the user on a movement direction of the user via the output function based on the movement course information along which the character has moved;
The control function is a virtual space movement program that designates a character based on the biometric information and movement information, and displays a character with a warning meaning when the biometric information becomes a physically abnormal value .

Images