JP7717628B2 - Information processing device - Google Patents

Description

The present invention relates to an information processing device.

Patent Document 1 discloses a technology for performing key operations using icons, which are virtual keys. With this technology, when three fingertips are detected, an icon is assigned to each of the three fingertip positions. If any of the three fingertips subsequently moves a distance equal to or greater than a predetermined value, the key operation corresponding to the icon assigned to the finger that moved the distance equal to or greater than the predetermined value is performed.

JP 2013-61848 A

In the technology described in Patent Document 1, a possible implementation is to use a gesture made with a finger (for example, a gesture of placing the index finger and middle finger together) as a trigger to start assigning a virtual key to a fingertip position. In this implementation, a virtual key is set at the fingertip position of the finger that made the gesture.

However, in this case, since the gesture is made with the finger that is assigned to the virtual key, it may be difficult to detect the position of the fingertip that made the gesture. This makes it easy for the virtual key to be assigned in a position different from the correct position where it should be assigned.

The object of the present invention is to provide an information processing device that can reduce the likelihood of virtual keys being set in a position other than the correct position.

An information processing device according to one embodiment includes a virtual key setting unit that positions a first virtual key in a virtual space; a gesture detection unit that detects a gesture made by at least one finger; a fingertip detection unit that detects the fingertip position of a first finger that is different from the at least one finger; and a virtual key control unit that, when the gesture detection unit detects the gesture while the first virtual key is located in the virtual space, moves the first virtual key to a target position in the virtual space that corresponds to the fingertip position of the first finger detected by the fingertip detection unit.

According to one aspect, it is possible to reduce the chance of virtual keys being set in a position other than the correct position.

FIG. 1 is a diagram illustrating an information processing system 100. FIG. 4 is a diagram showing an example of an image 4 representing a virtual keyboard 3. 1 is a diagram showing a right hand C of a user A and a virtual keyboard 3. FIG. FIG. 1 is a diagram showing an example of MR glasses 1. FIG. 2 is a diagram illustrating an example of a portable device 2. FIG. 2 is a diagram illustrating an example of the relationship between a world coordinate system CS2 and a local coordinate system CS1. FIG. 2 is a diagram showing an example of a virtual keyboard 3 located in a virtual space K2. FIG. 2 is a diagram showing an example of a reference plane P1 in a virtual space K2. FIG. 2 is a diagram showing an example of a virtual camera N1 placed in a virtual space K2. 10 is a diagram showing the positional relationship between a fingertip position C1 of a user A and a virtual keyboard 3. FIG. 10A and 10B are diagrams illustrating an example of movement of a virtual key 31H to a target position M1. 10A and 10B are diagrams for explaining an example of an operation of placing a virtual keyboard 3 in a virtual space K2. 10A and 10B are diagrams illustrating an example of an operation of moving the virtual keyboard 3. FIG. 10A and 10B are diagrams illustrating an example of movement of the virtual keyboard 3. FIG. 10 is a diagram showing an example of a portable device 2A according to a third modified example. 10 is a diagram showing an example of enlarging and displaying the virtual key that is closest to the target position M1 among the plurality of virtual keys 31. FIG.

A: First embodiment A1: Information processing system 100
FIG. 1 is a diagram showing an information processing system 100. The information processing system 100 includes MR (Mixed Reality) glasses 1 and a mobile device 2. The information processing system 100 displays a virtual keyboard 3 shown in FIG. 7 (described later) on the MR glasses 1. The virtual keyboard 3 is located in a virtual space K2. The form of the virtual keyboard 3 is not limited to the form shown in FIG. 7 and can be changed as appropriate.

User A wearing the MR glasses 1 shown in FIG. 1 can simultaneously view a virtual keyboard 3 located in virtual space K2 and a real object located in real space K1. The information processing system 100 accepts operations on the virtual keyboard 3 from user A. The information processing system 100 executes key operation processing in response to the operations on the virtual keyboard 3.

The MR glasses 1 are eyeglass-type display devices. The MR glasses 1 are an example of a transmissive display device. A transmissive display device is a display device that transmits light and displays an image. The transmissive display device is not limited to the MR glasses 1, and may also be, for example, a goggle-shaped transmissive HMD (Head Mounted Display).

A see-through display device is an example of a display device such as XR (X Reality) glasses. XR glasses are also called smart glasses. Display devices such as XR glasses are not limited to see-through display devices, and may be, for example, video see-through display devices. A video see-through display device has a camera that captures an object and generates an image of the object. A video see-through display device displays an image obtained by overlaying an image of a virtual object on the image generated by the camera. An example of the virtual object is a virtual keyboard 3. An example of a video see-through display device is a video see-through HMD.

MR glasses 1 include temples 91 and 92, a bridge 93, barrels 94 and 95, lenses 96L and 96R, and cameras 11L and 11R.

The body 94 includes a display panel for the left eye and an optical member for the left eye. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel.

The display panel for the left eye displays an image 4 representing a virtual keyboard 3 located in the virtual space K2. Figure 2 shows an example of the image 4 representing the virtual keyboard 3. The virtual keyboard 3 is a virtually generated keyboard. The virtual keyboard 3 does not exist in the real space K1. The virtual keyboard 3 has multiple virtual keys 31.

The left-eye display panel in the body 94 shown in FIG. 1 displays image 4 by emitting light representing image 4 of the virtual keyboard 3. The left-eye optical member in the body 94 is an optical member that guides the light emitted from the left-eye display panel to lens 96L.

The body 95 includes a display panel for the right eye and an optical element for the right eye. The display panel for the right eye displays an image 4 representing a virtual keyboard 3 located in the virtual space K2. The display panel for the right eye displays the image 4 by emitting light representing the image 4 of the virtual keyboard 3. The optical element for the right eye is an optical element that guides the light emitted from the display panel for the right eye to the lens 96R.

Lens 96L and 96R each have a half mirror. The half mirror of lens 96L reflects light guided by the optical element for the left eye to user A's left eye BL. The half mirror of lens 96L transmits light representing objects located in real space K1, thereby guiding the light representing objects located in real space K1 to user A's left eye BL. The half mirror of lens 96R reflects light guided by the optical element for the right eye to user A's right eye BR. The half mirror of lens 96R transmits light representing objects located in real space K1, thereby guiding the light representing objects located in real space K1 to user A's right eye BR. Therefore, user A wearing the MR glasses 1 can view user A's right hand C and the virtual keyboard 3, for example, as shown in FIG. 3.

Cameras 11L and 11R shown in FIG. 1 each capture an image of a subject. The subject is, for example, the left and right "hands" of user A. "Hand" refers to the body part of user A from the wrist to the fingertips. "Hand" includes the thumb, index finger, middle finger, ring finger, and little finger. In the following description, what is called a "thumb" in English may be called a "fingers." In other words, "fingers" is a concept that includes the thumb, index finger, middle finger, ring finger, and little finger.

Cameras 11L and 11R are separated from each other by a first distance, which is the baseline length. Therefore, when the subject is user A's left and right hands, information processing system 100 can identify the three-dimensional shapes of the left and right hands by using cameras 11L and 11R. The configuration for identifying the three-dimensional shapes of the left and right hands is not limited to cameras 11L and 11R. For example, the configuration for identifying the three-dimensional shapes of the left and right hands may be three or more cameras or a depth sensor. Cameras 11L and 11R are an example of two or more sensors that measure physical quantities to identify the three-dimensional shapes of the left and right hands.

The three-dimensional shapes of the left and right hands indicate the fingertip positions of the ten fingers and the shapes of the left and right palms. Therefore, the information processing system 100 can detect gestures made by the fingers of either the left or right hand based on changes in the three-dimensional shapes of the left and right hands.

A local coordinate system CS1 is defined for the MR glasses 1. The local coordinate system CS1 is the coordinate system of the MR glasses 1. The three-dimensional shapes of the left and right hands identified using cameras 11L and 11R are indicated by the coordinates of the local coordinate system CS1. The local coordinate system CS1 is defined by the x1 axis, y1 axis, and z1 axis. The x1 axis, y1 axis, and z1 axis are mutually orthogonal. The origin of the local coordinate system CS1 is located at camera 11L. The z1 axis coincides with the imaging direction of camera 11L.

In FIG. 1, for simplicity of explanation, the origin of the local coordinate system CS1 is not located at camera 11L. The z1 axis does not coincide with the imaging direction of camera 11L. Note that, as shown in FIG. 1, the origin of the local coordinate system CS1 does not have to be located at camera 11L. The z1 axis does not have to coincide with the imaging direction of camera 11L. The origin of the local coordinate system CS1 may be located at camera 11R instead of camera 11L. If the origin of the local coordinate system CS1 is located at camera 11R, the z1 axis may coincide with the imaging direction of camera 11R.

The mobile device 2 is a smartphone. The mobile device 2 is not limited to a smartphone and may be, for example, a tablet or notebook personal computer. The mobile device 2 is also referred to as a terminal device. The mobile device 2 is an example of an information processing device. The information processing device is not limited to the mobile device 2 and may be, for example, a desktop personal computer that communicates directly or indirectly with the MR glasses 1, or a server that communicates directly or indirectly with the MR glasses 1.

The portable device 2 is connected to the MR glasses 1 via a wired connection. The portable device 2 may also be connected to the MR glasses 1 wirelessly. The portable device 2 controls the MR glasses 1. For example, the portable device 2 causes the MR glasses 1 to display an image 4 representing a virtual keyboard 3.

A2: MR Glass 1
4 is a diagram showing an example of the MR glasses 1. In addition to the cameras 11L and 11R, the MR glasses 1 include a display device 12, an acceleration sensor 13, a gyro sensor 14, operation buttons 15, a communication device 16, a storage device 17, a processing device 18, and a bus 19.

The bus 19 is a wiring for communicating information. The bus 19 interconnects the camera 11L, the camera 11R, the display device 12, the acceleration sensor 13, the gyro sensor 14, the operation buttons 15, the communication device 16, the storage device 17, and the processing device 18. The bus 19 may be configured using a single bus, or may be configured using different buses between each element of the device, etc.

Camera 11L generates image data D1 by capturing an image of a subject. Image data D1 is data representing a moving image. Image data D1 includes a series of still image data. Each still image data in image data D1 represents a still image that constitutes one frame of the moving image represented by image data D1.

Camera 11R captures an image of a subject and outputs image data D2. Image data D2 is data representing a moving image. Image data D2 includes a series of still image data. Each still image data in image data D2 represents a still image that constitutes one frame of the moving image represented by image data D2.

The display device 12 includes the lens 96L, the display panel for the left eye, the optical element for the left eye, the lens 96R, the display panel for the right eye, and the optical element for the right eye, as described with reference to FIG. 1. The display device 12 transmits light representing an object located in real space K1. The display device 12 also displays an image 4 representing a virtual keyboard 3 located in virtual space K2. When user A wears the MR glasses 1, the display device 12 is located in front of user A's left eye BL and right eye BR. Therefore, user A wearing the MR glasses 1 can see objects located in real space K1 and objects located in virtual space K2. For example, user A can see user A's right hand C located in real space K1 and the virtual keyboard 3 located in virtual space K2.

The acceleration sensor 13 measures the acceleration of the MR glasses 1. The acceleration sensor 13 generates acceleration data E1 based on the measurement results of the acceleration of the MR glasses 1. The acceleration data E1 is data indicating the acceleration of the MR glasses 1. For example, the acceleration data E1 indicates the acceleration of the MR glasses 1 in each of the x1-axis, y1-axis, and z1-axis directions. Based on the acceleration data E1, the movement direction and movement distance of the MR glasses 1 are determined.

The gyro sensor 14 measures the angular acceleration of the rotation of the MR glass 1 around each of the x1, y1, and z1 axes. The gyro sensor 14 generates angular acceleration data E2 based on the measurement results of the angular acceleration of the MR glass 1. The angular acceleration data E2 is data that indicates the angular acceleration of the MR glass 1. For example, the angular acceleration data E2 indicates the angular acceleration of the rotation of the MR glass 1 around each of the x1, y1, and z1 axes. The rotation of the MR glass 1 is determined based on the angular acceleration data E2.

Operation button 15 is a button that can be operated by user A. Operation button 15 may have multiple buttons that can be operated by user A. Operation button 15 generates operation data F1. Operation data F1 is data that indicates the operation state of operation button 15.

The communication device 16 communicates with the portable device 2 via a wired connection. The communication device 16 may also communicate with the portable device 2 wirelessly. The communication device 16 transmits comprehensive data G1 to the portable device 2. The comprehensive data G1 is data that includes image data D1, image data D2, acceleration data E1, angular acceleration data E2, and operation data F1. The comprehensive data G1 does not have to include operation data F1. The communication device 16 receives image data H1 from the portable device 2. The image data H1 is data that represents an image 4 of the virtual keyboard 3. The image data H1 is generated by the portable device 2, for example, based on the image data D1, image data D2, acceleration data E1, and angular acceleration data E2.

The storage device 17 is a recording medium readable by the processing device 18. The storage device 17 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory). The storage device 17 stores the program PG1.

The processing device 18 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

The processing device 18 reads the program PG1 from the storage device 17. By executing the program PG1, the processing device 18 functions as an acquisition unit 181 and an operation control unit 182. At least one of the acquisition unit 181 and the operation control unit 182 may be configured by a circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

The acquisition unit 181 acquires image data D1, image data D2, acceleration data E1, angular acceleration data E2, and operation data F1.

The operation control unit 182 controls the MR glasses 1. For example, the operation control unit 182 receives image data D1, image data D2, acceleration data E1, angular acceleration data E2, and operation data F1 from the acquisition unit 181. The operation control unit 182 generates comprehensive data G1 including the image data D1, image data D2, acceleration data E1, angular acceleration data E2, and operation data F1. The operation control unit 182 causes the communication device 16 to transmit the comprehensive data G1 to the mobile device 2. When the communication device 16 receives image data H1 from the mobile device 2, the operation control unit 182 acquires the image data H1 from the communication device 16. The operation control unit 182 causes the display device 12 to display an image based on the image data H1. The image based on the image data H1 is, for example, image 4 representing a virtual keyboard 3.

A3: Portable device 2
5 is a diagram showing an example of the portable device 2. The portable device 2 includes an input device 21, a display device 22, a communication device 23, a storage device 24, a processing device 25, and a bus 26.

The bus 26 is a wiring for communicating information. The bus 26 interconnects the input device 21, the display device 22, the communication device 23, the storage device 24, and the processing device 25. The bus 26 may be configured using a single bus, or may be configured using different buses between each element of the device, etc.

The input device 21 includes a touch panel. In addition to the touch panel, the input device 21 may also include multiple operation keys. The input device 21 may also include multiple operation keys without including a touch panel. The input device 21 accepts operations performed by user A.

The display device 22 includes a display. The touch panel of the input device 21 is layered on top of the display of the display device 22. The display device 22 displays various information.

The communication device 23 communicates with the MR glasses 1 via a wired connection. The communication device 23 may also communicate with the MR glasses 1 wirelessly. The communication device 23 receives comprehensive data G1 from the MR glasses 1. The communication device 23 transmits image data H1 to the MR glasses 1.

The storage device 24 is a recording medium readable by the processing device 25. The storage device 24 includes, for example, non-volatile memory and volatile memory. The storage device 24 stores the program PG2.

The processing device 25 includes one or more CPUs. The processing device 25 is another example of an information processing device. The processing device 25 reads the program PG2 from the storage device 24. By executing the program PG2, the processing device 25 functions as a hand data generation unit 251, a fingertip detection unit 252, a coordinate conversion unit 253, a virtual key setting unit 254, an image data generation unit 255, a gesture detection unit 256, a virtual key control unit 257, and an operation detection unit 258. At least one of the hand data generation unit 251, the fingertip detection unit 252, the coordinate conversion unit 253, the virtual key setting unit 254, the image data generation unit 255, the gesture detection unit 256, the virtual key control unit 257, and the operation detection unit 258 may be configured using a circuit such as a DSP, an ASIC, or an FPGA.

The hand data generation unit 251 acquires the comprehensive data G1 via the communication device 23. The hand data generation unit 251 reads the image data D1 and D2 from the comprehensive data G1. The image data D1 is image data generated by the camera 11L of the MR glasses 1. The image data D2 is image data generated by the camera 11R of the MR glasses 1. The hand data generation unit 251 generates hand data J1 based on the image data D1 and D2. The hand data J1 is data that indicates the three-dimensional shape of both hands of user A using coordinates in the local coordinate system CS1 defined in the MR glasses 1.

For example, the hand data generation unit 251 generates hand data J1 based on the still image data included in image data D1 and the still image data included in image data D2. The hand data generation unit 251 changes each of the still image data in image data D1 used to generate hand data J1 and the still image data in image data D2 used to generate hand data J1 to the latest still image data every time a predetermined time elapses. Each time the hand data generation unit 251 changes the still image data, it generates hand data J1 using the latest still image data. The predetermined time is, for example, 0.2 seconds. The predetermined time is not limited to 0.2 seconds and may be shorter or longer than 0.2 seconds.

The fingertip detection unit 252 detects the fingertips of each finger on user A's left and right hands based on the hand data J1. The fingertip detection unit 252 generates fingertip data J2 based on the detection results of the fingertips of each finger. The fingertip data J2 is data that indicates the fingertip position of each finger using coordinates in the local coordinate system CS1 defined in the MR glasses 1. The fingertip detection unit 252 generates fingertip data J2 for each piece of hand data J1.

The coordinate conversion unit 253 converts the coordinates of the local coordinate system CS1 defined in the MR glass 1 into coordinates of the world coordinate system CS2. The world coordinate system CS2 is a coordinate system defined in the real space K1.

Figure 6 shows an example of the relationship between the world coordinate system CS2 and the local coordinate system CS1. The world coordinate system CS2 is defined by the x2 axis, y2 axis, and z2 axis. The x2 axis, y2 axis, and z2 axis are mutually orthogonal.

The coordinate conversion unit 253 shown in FIG. 5 determines the positions of the x2, y2, and z2 axes of the world coordinate system CS2 in the real space K1 to the positions of the x1, y1, and z1 axes of the local coordinate system CS1 in the real space K1 at timing t0. Timing t0 refers to the timing for determining the world coordinate system CS2 in the real space K1. Timing t0 is, for example, the timing at which the coordinate conversion unit 253 receives a "keyboard setting instruction" input by user A to the input device 21. The "keyboard setting instruction" is an instruction for setting up the virtual keyboard 3.

At time t0, the x1, y1, and z1 axes of the local coordinate system CS1 coincide with the x2, y2, and z2 axes of the world coordinate system CS2, respectively. As the MR glass 1 moves after time t0, the x1, y1, and z1 axes of the local coordinate system CS1 deviate from the x2, y2, and z2 axes of the world coordinate system CS2, respectively. As the MR glass 1 rotates after time t0, the x1, y1, and z1 axes of the local coordinate system CS1 deviate from the x2, y2, and z2 axes of the world coordinate system CS2, respectively.

The coordinate conversion unit 253 reads the acceleration data E1 from the comprehensive data G1. Based on the acceleration data E1, the coordinate conversion unit 253 determines the movement of the MR glass 1 after time t0. The coordinate conversion unit 253 determines the movement of the MR glass 1 as the direction of movement of the MR glass 1 after time t0 and the distance of movement of the MR glass 1 after time t0.

The coordinate conversion unit 253 reads the angular acceleration data E2 from the comprehensive data G1. Based on the angular acceleration data E2, the coordinate conversion unit 253 determines the rotation of the MR glass 1 after time t0. The coordinate conversion unit 253 determines the rotation direction of the MR glass 1 after time t0 and the rotation angle of the MR glass 1 after time t0 as the rotation of the MR glass 1.

The coordinate conversion unit 253 identifies the movement and rotation of the MR glass 1 each time it receives fingertip data J2 from timing t0 onwards. Each time it identifies the movement and rotation of the MR glass 1, the coordinate conversion unit 253 converts the fingertip position of each finger indicated by the fingertip data J2 from coordinates in the local coordinate system CS1 to coordinates in the world coordinate system CS2 based on the most recent movement and rotation of the MR glass 1.

For example, the coordinate conversion unit 253 identifies a translation matrix corresponding to the movement of the MR glass 1 based on the movement of the MR glass 1 during the period from timing t0 to the time when the latest fingertip data J2 is received. The coordinate conversion unit 253 identifies a rotation matrix corresponding to the rotation of the MR glass 1 based on the rotation of the MR glass 1 during the period from timing t0 to the time when the latest fingertip data J2 is received. The coordinate conversion unit 253 uses the translation matrix and the rotation matrix to convert the fingertip position of each finger indicated by the latest fingertip data J2 from coordinates in the local coordinate system CS1 to coordinates in the world coordinate system CS2. The coordinate conversion unit 253 generates fingertip data J3. The fingertip data J3 is data indicating the fingertip position of each finger indicated by the latest fingertip data J2 using coordinates in the world coordinate system CS2. The coordinate conversion unit 253 generates fingertip data J3 for each piece of fingertip data J2.

In addition, each time the movement and rotation of the MR glass 1 are identified, the coordinate conversion unit 253 converts the coordinates of the local coordinate system CS1 of the camera 11L into coordinates of the world coordinate system CS2 based on the latest movement and rotation of the MR glass 1. The coordinates of the local coordinate system CS1 of the camera 11L are determined in advance. The coordinates of the local coordinate system CS1 of the camera 11L are, for example, the coordinates of the origin of the local coordinate system CS1.

Each time the movement and rotation of the MR glass 1 are identified, the coordinate conversion unit 253 converts the coordinates of the local coordinate system CS1 indicating the imaging direction of the camera 11L into coordinates of the world coordinate system CS2 based on the latest movement and rotation of the MR glass 1. The imaging direction of the camera 11L is determined in advance. The imaging direction of the camera 11L is, for example, the direction of the x1 axis of the local coordinate system CS1.

The coordinate conversion unit 253 generates camera data J4. The camera data J4 is data indicating the coordinates of the camera 11L in the world coordinate system CS2 and the coordinates of the world coordinate system CS2 that indicate the imaging direction of the camera 11L. The camera data J4 is not limited to the above data, and may also be data indicating the coordinates of the camera 11R in the world coordinate system CS2 and the coordinates of the world coordinate system CS2 that indicate the imaging direction of the camera 11R. The camera data J4 may also be data indicating the coordinates of the world coordinate system CS2 of a point between the cameras 11L and 11R and the coordinates of the world coordinate system CS2 that indicate the imaging direction of the camera 11L or 11R. The camera data J4 is used, for example, to generate image data H1 that represents an image 4 of the virtual keyboard 3. The coordinate conversion unit 253 generates camera data J4 each time the movement and rotation of the MR glasses 1 are identified.

The virtual key setting unit 254 positions the virtual keyboard 3 in the virtual space K2. FIG. 7 is a diagram showing an example of the virtual keyboard 3 positioned in the virtual space K2. The virtual space K2 is a virtual three-dimensional space generated by the virtual key setting unit 254. A world coordinate system CS2 is defined in the virtual space K2. In the virtual space K2, the positions of the x2 axis, y2 axis, and z2 axis of the world coordinate system CS2 are fixed in advance.

The virtual key setting unit 254 shown in FIG. 5 positions the virtual keyboard 3 in the virtual space K2 in accordance with timing t1. Timing t1 refers to the timing for positioning the virtual keyboard 3 in the virtual space K2. Timing t1 is, for example, the timing at which the virtual key setting unit 254 receives a "keyboard setting instruction" input by user A to the input device 21.

Timing t1 is not limited to the above-mentioned timing. For example, timing t1 may be the timing when the gesture detection unit 256 (described later) detects a first gesture performed by user A. The first gesture is, for example, a gesture in which the tip of the index finger of user A's left hand touches the tip of the thumb of user A's left hand. The first gesture is not limited to the above-mentioned gesture. For example, the first gesture may be a gesture in which the tip of the middle finger of user A's left hand touches the tip of the thumb of user A's left hand.

The virtual key setting unit 254 determines the reference plane P1 on which the virtual keyboard 3 is placed within the virtual space K2 according to timing t1.

Figure 8 is a diagram showing an example of the reference plane P1 in the virtual space K2. In Figure 8, the right hand C of user A, which does not exist in the virtual space K2, is shown with a dotted line for illustrative purposes. The coordinate position of the right hand C shown in Figure 8 in the world coordinate system CS2 in the virtual space K2 coincides with the coordinate position of the right hand C in the world coordinate system CS2 in the real space K1.

The virtual key setting unit 254 shown in FIG. 5 determines the reference plane P1 based on, for example, the fingertip positions of multiple fingers on the right hand C. At least three fingertip positions are required to determine the reference plane P1.

The virtual key setting unit 254 first identifies the fingertip positions of three of the five fingers on the right hand C based on the latest fingertip data J3 at time t1. The virtual key setting unit 254 determines the plane including the fingertip positions of the three fingers as the reference plane P1.

Once three points are determined in virtual space K2, the plane containing these three points is uniquely determined. If the reference plane P1 is a plane, the virtual key setting unit 254 determines the plane containing the fingertip positions of the three fingers as the reference plane P1.

The reference plane P1 is not limited to a flat surface, and may be, for example, a curved surface. When the reference plane P1 is a curved surface, the virtual key setting unit 254 determines the curved surface using a predetermined function. The predetermined function uniquely defines a curved surface according to the three-dimensional coordinates of the three input points. The virtual key setting unit 254 inputs the fingertip positions of three fingers into the predetermined function. The virtual key setting unit 254 determines the curved surface defined by the predetermined function according to the fingertip positions of the three fingers as the reference plane P1. When the reference plane P1 is a curved surface, the virtual keyboard 3 may be a so-called ergonomic keyboard.

The fingertip positions of the three fingers used to determine reference plane P1 are, for example, the fingertip positions of the thumb, ring finger, and little finger. When viewed from the back of the right hand C, the thumb, ring finger, and little finger are located at the left end of the right hand C, a position to the left of the right end of the right hand C, and the right end of the right hand C, respectively. Therefore, when the fingertip positions of the thumb, ring finger, and little finger are used to determine reference plane P1, reference plane P1 is determined by the entire right hand C.

The virtual key setting unit 254 may determine the reference plane P1 based on the fingertip positions of four or more fingers. In this case, the virtual key setting unit 254 first identifies the distance from the fingertip position of each finger to the reference plane P1 as the error. Next, the virtual key setting unit 254 determines the reference plane P1 that minimizes the total error, for example, using the least squares method.

The virtual key setting unit 254 may determine the reference plane P1 without using the fingertip positions of multiple fingers. For example, the virtual key setting unit 254 may determine the reference plane P1 at any position in the virtual space K2.

Once the virtual key setting unit 254 determines the reference plane P1, it places the virtual keyboard 3 on the reference plane P1.

When the virtual keyboard 3 is placed on the reference plane P1, the virtual key setting unit 254 identifies the fingertip position of the middle finger of the right hand C based on the latest fingertip data J3. The middle finger of the right hand C is an example of a reference finger. The reference finger is not limited to the middle finger of the right hand C, and may be, for example, the thumb of the right hand C, the index finger of the right hand C, the ring finger of the right hand C, or the little finger of the right hand C.

When the virtual key setting unit 254 identifies the fingertip position of the middle finger of the right hand C, it shifts the virtual keyboard 3 along the reference plane P1 to position the virtual key 31 representing "H" on the virtual keyboard 3 at the fingertip position of the middle finger of the right hand C. Hereinafter, the virtual key 31 representing "H" will be referred to as virtual key 31H. Virtual key 31H is an example of a first virtual key. The first virtual key is not limited to virtual key 31H, and may be a virtual key 31 different from virtual key 31H. A virtual key 31 different from virtual key 31H is, for example, the virtual key 31 representing "J" or the virtual key 31 representing "K."

When the virtual key 31H is positioned at the tip of the middle finger of the right hand C in the virtual space K2, the virtual key setting unit 254 generates virtual space data J5. The virtual space data J5 is data representing the virtual space K2 in which the virtual keyboard 3 is placed.

The image data generation unit 255 generates image data H1 based on the virtual space data J5 and the camera data J4. The camera data J4 indicates the coordinates of the camera 11L in the world coordinate system CS2 and the coordinates of the world coordinate system CS2 that indicate the imaging direction of the camera 11L.

The image data generation unit 255 places the virtual camera N1 in the virtual space K2 represented by the virtual space data J5 at the coordinate position in the world coordinate system CS2 of camera 11L indicated by the camera data J4. When the camera data J4 indicates the coordinate position in the world coordinate system CS2 of camera 11R, the image data generation unit 255 places the virtual camera N1 in the virtual space K2 at the coordinate position in the world coordinate system CS2 of camera 11R indicated by the camera data J4. When the camera data J4 indicates the coordinate position in the world coordinate system CS2 of a point between cameras 11L and 11R, the image data generation unit 255 places the virtual camera N1 in the virtual space K2 at the coordinate position in the world coordinate system CS2 of the point indicated by the camera data J4.

Figure 9 shows an example of a virtual camera N1 placed in virtual space K2. The imaging direction of virtual camera N1 matches the imaging direction of camera 11L indicated by camera data J4. When camera data J4 indicates the imaging direction of camera 11R, the imaging direction of virtual camera N1 matches the imaging direction of camera 11R indicated by camera data J4.

The image data generation unit 255 shown in FIG. 5 generates captured image data obtained by the virtual camera N1 capturing an image of the virtual space K2 as image data H1. Therefore, the image data H1 represents an image corresponding to the line of sight of user A wearing the MR glasses 1. The image corresponding to the line of sight of user A wearing the MR glasses 1 is, for example, image 4 representing the virtual keyboard 3.

The image data generation unit 255 causes the communication device 23 to transmit the image data H1 to the MR glasses 1. When the MR glasses 1 receive the image data H1, they display the image represented by the image data H1 (for example, image 4 of the virtual keyboard 3). This allows user A to operate the virtual keyboard 3 while looking at the virtual keyboard 3 displayed on the MR glasses 1.

As time passes from the point in time when the virtual keyboard 3 is placed in the virtual space K2, it becomes more difficult for user A to recognize the distance between user A's fingers and the virtual keyboard 3. For example, if user A's fingertip position C1 moves away from the virtual keyboard 3 over time, as shown in FIG. 10, user A will find it more difficult to recognize the distance between user A's fingers and the virtual keyboard 3.

The portable device 2 can reset the positional relationship between user A's fingers and the virtual keyboard 3 so that user A can once again recognize the distance between user A's fingers and the virtual keyboard 3. For example, when the portable device 2 detects a second gesture performed by a finger on user A's left hand, it moves the virtual key 31H to target position M1 in the virtual space K2 corresponding to the fingertip position of the middle finger on user A's right hand C. In other words, the portable device 2 moves the virtual key 31H to target position M1 corresponding to the fingertip position of a finger that will not perform the second gesture. In this way, on the portable device 2, the finger that performs the second gesture is different from the finger that corresponds to target position M1.

Note that if the finger performing the second gesture is the same as the finger corresponding to target position M1, it may be difficult to detect the fingertip position of the finger corresponding to target position M1, depending on the content of the second gesture. Furthermore, to make it easier to detect the fingertip position of the finger corresponding to target position M1 when the finger performing the second gesture is the same as the finger corresponding to target position M1, the content of the second gesture is limited. In other words, the degree of freedom of the second gesture is reduced.

On mobile device 2, the finger performing the second gesture is different from the finger corresponding to target position M1. This reduces the difficulty in detecting the fingertip position of the finger corresponding to target position M1 due to the second gesture, and prevents the degree of freedom of the second gesture from being reduced.

Figure 11 is a diagram showing an example of movement of a virtual key 31H to a target position M1. In Figure 11, the virtual keyboard 3 shown by dotted lines and the virtual key 31H shown by dotted lines represent the virtual keyboard 3 and the virtual key 31H before movement. The virtual keyboard 3 shown by solid lines and the virtual key 31H shown by solid lines represent the virtual keyboard 3 after movement and the virtual key 31H before movement.

The gesture detection unit 256 shown in FIG. 5 detects a second gesture. The second gesture is a gesture for resetting the positional relationship between user A's fingers and the virtual keyboard 3. The second gesture is, for example, a gesture in which user A's index finger on his left hand touches the middle finger on his left hand. The index finger on user A's left hand and the middle finger on user A's left hand are examples of at least one finger. At least one finger is one, two, three, four, five, six, seven, eight, or nine of the ten fingers on both hands of user A.

The second gesture is not limited to the gestures described above. For example, the second gesture may be a gesture in which user A touches the tip of the little finger on his or her left hand to the tip of the thumb on his or her left hand. The second gesture may be the same as the first gesture.

The gesture detection unit 256 detects the second gesture based on multiple pieces of hand data J1, including the latest hand data J1. The hand data J1 indicates the three-dimensional shape of both hands of user A. When the gesture detection unit 256 detects the second gesture, it generates detection data J6. The detection data J6 is data representing the detection of the second gesture. The gesture detection unit 256 provides the detection data J6 to the virtual key control unit 257.

The virtual key control unit 257 resets the positional relationship between user A's fingers and the virtual keyboard 3. If the gesture detection unit 256 detects a second gesture when the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 moves the virtual key 31H to the target position M1 in the virtual space K2.

When the virtual key control unit 257 receives detection data J6, it determines that the gesture detection unit 256 has detected the second gesture.

When the virtual key control unit 257 receives the detection data J6, it determines whether the virtual keyboard 3 is located in the virtual space K2 based on the virtual space data J5. For example, the virtual key control unit 257 determines whether the virtual keyboard 3 is located in the virtual space K2 by referencing the virtual space data J5. If the virtual space data J5 represents the virtual keyboard 3, the virtual key control unit 257 determines that the virtual keyboard 3 is located in the virtual space K2. If the virtual space data J5 does not represent the virtual keyboard 3, the virtual key control unit 257 determines that the virtual keyboard 3 is not located in the virtual space K2.

The virtual space data J5 is generated when the virtual key setting unit 254 positions the virtual key 31H in the virtual space K2. Therefore, if the virtual space data J5 has been generated, the virtual key control unit 257 may determine that the virtual keyboard 3 is located in the virtual space K2. If the virtual space data J5 has not been generated, the virtual key control unit 257 may determine that the virtual keyboard 3 is not located in the virtual space K2.

When the virtual key control unit 257 determines that the virtual keyboard 3 is located in the virtual space K2, it determines the fingertip position of the middle finger of the right hand C indicated by the latest fingertip data J3 as the target position M1.

The middle finger of the right hand C is an example of a first finger that is different from "at least one finger." The first finger is not limited to the middle finger of the right hand C, and may be, for example, the thumb of the right hand C, the index finger of the right hand C, the ring finger of the right hand C, or the little finger of the right hand C. The first finger may also be the reference finger described above. The reference finger is a finger used to position the virtual keyboard 3 in the virtual space K2. In a configuration in which the first finger is the reference finger, user A can recognize the positional relationship between the virtual keyboard 3 and the user's fingers by being aware of the positional relationship between the first finger and one finger used as the reference finger and the virtual keyboard 3. Therefore, compared to a configuration in which the first finger is different from the reference finger, user A can more easily recognize the positional relationship between the position of the first virtual key and the user's finger. Note that the first finger may be different from the reference finger.

The fingertip position of the middle finger of the right hand C indicated by the latest fingertip data J3 is an example of a "target position in virtual space K2 corresponding to the fingertip position of the first finger detected by the fingertip detection unit 252."

Once the virtual key control unit 257 determines the target position M1, it controls the virtual key setting unit 254 to move the virtual key 31H to the target position M1 in the virtual space K2.

For example, the virtual key control unit 257 moves the virtual keyboard 3 to move the virtual key 31H to the target position M1. As an example, the virtual key control unit 257 provides a first movement instruction J7 to the virtual key setting unit 254, thereby causing the virtual key setting unit 254 to move the virtual keyboard 3 and move the virtual key 31H to the target position M1. The first movement instruction J7 is an instruction to move the virtual keyboard 3 to move the virtual key 31H to the target position M1. In response to the first movement instruction J7, the virtual key setting unit 254 moves the virtual keyboard 3 in the virtual space K2 to move the virtual key 31H to the target position M1.

If the virtual keyboard 3 is not located in the virtual space K2, the virtual key control unit 257 does not move the virtual key 31H.

The virtual key control unit 257 may cause the virtual key setting unit 254 to redetermine the reference plane P1 based on the latest fingertip data J3 during the period from determining the target position M1 to providing the first movement instruction J7. For example, the virtual key control unit 257 provides the virtual key setting unit 254 with an instruction to redetermine the reference plane P1 during the period from determining the target position M1 to providing the first movement instruction J7. The method for redetermining the reference plane P1 is the same as the method for determining the reference plane P1. In this case, the virtual key control unit 257 moves the virtual keyboard 3 on the redetermined reference plane P1, thereby moving the virtual key 31H to the target position M1.

When the virtual keyboard 3 is moved, the virtual key setting unit 254 updates the virtual space data J5. The updated virtual space data J5 represents the virtual space K2 in which the moved virtual keyboard 3 is located.

The operation detection unit 258 detects operations on the virtual keyboard 3. For example, the operation detection unit 258 detects operations on the virtual keyboard 3 based on the latest virtual space data J5 and the latest fingertip data J3. For example, if one of the multiple fingertip positions indicated by the latest fingertip data J3 falls within one of the virtual keys 31 on the virtual keyboard 3, the operation detection unit 258 detects an operation on the virtual key 31 where the fingertip position falls. The operation detection unit 258 executes key operation processing based on the detection result of the operation on the virtual keyboard 3. The key operation processing is processing associated with the operated virtual key 31. For example, if virtual key 31H is operated, the operation detection unit 258 executes processing to input the letter "H" as the key operation processing.

A4: Description of Operation The operation of the portable device 2 will be described. First, the operation of placing the virtual keyboard 3 in the virtual space K2 will be described. Next, the operation of moving the virtual keyboard 3 will be described.

Figure 12 is a diagram illustrating an example of the operation of placing the virtual keyboard 3 in the virtual space K2.

In step S101, the hand data generation unit 251 generates hand data J1 based on image data D1 and D2. Hand data J1 is data that indicates the three-dimensional shape of both hands of user A using coordinates in local coordinate system CS1.

Next, in step S102, the fingertip detection unit 252 generates fingertip data J2 based on the hand data J1. The fingertip data J2 is data that indicates the fingertip positions of each finger on user A's left and right hands using coordinates in the local coordinate system CS1.

Next, in step S103, the coordinate conversion unit 253 generates fingertip data J3 based on the fingertip data J2, acceleration data E1, and angular acceleration data E2. The fingertip data J3 is data that indicates the fingertip position of each finger indicated by the fingertip data J2 using coordinates in the world coordinate system CS2.

Next, in step S104, the coordinate conversion unit 253 generates camera data J4 based on the acceleration data E1 and angular acceleration data E2. The camera data J4 is data indicating the coordinates of the camera 11L in the world coordinate system CS2 and the coordinates of the world coordinate system CS2 that indicate the imaging direction of the camera 11L. Step S104 may be executed before step S103 is executed.

Next, in step S105, the virtual key setting unit 254 positions the virtual keyboard 3 in the virtual space K2 at timing t1. Timing t1 is, for example, the timing at which the virtual key setting unit 254 receives a "keyboard setting instruction" input by user A to the input device 21.

In step S105, the virtual key setting unit 254 determines a reference plane P1 within the virtual space K2, for example, based on the fingertip positions of multiple fingers of the right hand C. Once the virtual key setting unit 254 determines the reference plane P1, it places a virtual keyboard 3 on the reference plane P1. Once the virtual key setting unit 254 places the virtual keyboard 3 on the reference plane P1, it identifies the fingertip position of the middle finger of the right hand C based on the latest fingertip data J3. Once the virtual key setting unit 254 identifies the fingertip position of the middle finger of the right hand C, it shifts the virtual keyboard 3 along the reference plane P1 to position the virtual key 31H representing "H" at the fingertip position of the middle finger of the right hand C.

Figure 13 is a diagram illustrating an example of an operation for moving the virtual keyboard 3. When the gesture detection unit 256 detects a second gesture based on the hand data J1 in step S201, it generates detection data J6 in step S202. The second gesture is, for example, a gesture in which the index finger of user A's left hand touches the middle finger of user A's left hand. The detection data J6 is data representing the detection of the second gesture.

Next, in step S203, the virtual key control unit 257 determines whether the virtual keyboard 3 is located in the virtual space K2 based on the virtual space data J5.

When the virtual key control unit 257 determines in step S203 that the virtual keyboard 3 is located in the virtual space K2, it determines the target position M1 based on the fingertip data J3 in step S204. The target position M1 is the fingertip position of the middle finger of the right hand C indicated by the most recent fingertip data J3.

Next, in step S205, the virtual key control unit 257 controls the virtual key setting unit 254 to move the virtual key 31H to the target position M1 in the virtual space K2. As a result, user A wearing the MR glasses 1 feels as if the virtual keyboard 3 is sticking to the tip of the middle finger of his right hand C.

If the gesture detection unit 256 does not detect the second gesture in step S201, the processing shown in FIG. 13 ends. If the virtual keyboard 3 is not located in the virtual space K2 in step S203, the processing shown in FIG. 13 ends.

A5: Summary of First Embodiment The virtual key setting unit 254 positions the virtual key 31H in the virtual space K2. The gesture detection unit 256 detects a second gesture in which the index finger of user A's left hand touches the middle finger of user A's left hand. The fingertip detection unit 252 detects the fingertip positions of each finger on user A's left and right hands. Therefore, the fingertip detection unit 252 detects at least the fingertip position of the middle finger of user A's right hand C. When the gesture detection unit 256 detects the second gesture in a situation in which the virtual key 31H is located in the virtual space K2, the virtual key control unit 257 moves the virtual key 31H to a target position M1 in the virtual space K2 that corresponds to the fingertip position of the middle finger of user A's right hand C detected by the fingertip detection unit 252.

The gesture detection unit 256 detects gestures made with a finger other than the middle finger of the right hand C that corresponds to the destination of virtual key 31H. This prevents a situation in which a gesture is made with the middle finger of the right hand C that corresponds to the destination of virtual key 31H. This solves the problem of it being difficult to detect the fingertip position of the middle finger of user A's right hand C due to gestures made with the middle finger of the right hand C.

Virtual key 31H is an example of a first virtual key. The first virtual key is one of the virtual keys 31 included in the virtual keyboard 3. The first virtual key is not limited to one of the virtual keys 31 included in the virtual keyboard 3. For example, the first virtual key may be a virtual key that is not included in the virtual keyboard 3. A virtual key that is not included in the virtual keyboard 3 is, for example, a virtual PLAY key for videos or a virtual STOP key for videos.

The middle finger of the right hand C, which corresponds to the destination of virtual key 31H, is an example of the first finger. The first finger is not limited to the middle finger of the right hand C. The first finger may be any one of the ten fingers on the left or right hand of user A.

The index finger and middle finger of user A's left hand that perform the second gesture are an example of at least one finger that performs the second gesture. The at least one finger that performs the second gesture is not limited to the index finger and middle finger of user A's left hand. The at least one finger that performs the second gesture may be one or more fingers other than the first finger out of the ten fingers on user A's left and right hands.

In the first embodiment, the combination of the finger performing the second gesture and the first finger is a combination of a finger on the left hand and a finger on the right hand C. A combination of a finger on the left hand and a finger on the right hand C means a combination of fingers on different hands. The combination of the finger performing the second gesture and the first finger is not limited to a combination of fingers on different hands. For example, the combination of the finger performing the second gesture and the first finger may be a combination of different fingers on the same hand. An example of a combination of different fingers on the same hand is a combination of the thumb of the right hand C and the index finger of the right hand C, or a combination of the index finger of the left hand and the middle finger of the left hand.

The virtual key setting unit 254 positions the virtual keyboard 3 in the virtual space K2, thereby positioning the virtual key 31H in the virtual space K2. If the gesture detection unit 256 detects a second gesture when the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 moves the virtual key 31H to the target position M1. This allows the virtual keys included in the virtual keyboard 3 to be moved to the target position M1.

When the gesture detection unit 256 detects a second gesture while the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 moves the virtual keyboard 3 to move the virtual key 31H to the target position M1. This allows the entire virtual keyboard 3 to move in response to the second gesture.

B: Modifications Modifications of the above-described embodiment are shown below. Two or more of the following modifications may be arbitrarily selected and combined as long as they are not mutually contradictory.

B1: First Modification In the first embodiment, instead of moving the first virtual key (e.g., virtual key 31H) to the target position M1 by moving the virtual keyboard 3, the virtual key control unit 257 may move the first virtual key to the target position M1 without moving any virtual key 31 among the multiple virtual keys 31 that is different from the first virtual key.

For example, when the gesture detection unit 256 detects a second gesture when the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 provides a second movement instruction to the virtual key setting unit 254. The second movement instruction is an instruction to move the first virtual key 31 to the target position M1 without moving any virtual keys 31 of the multiple virtual keys 31 that are different from the first virtual key (e.g., virtual key 31H). In response to the second movement instruction, the virtual key setting unit 254 moves the first virtual key 31 to the target position M1 in the virtual space K2 without moving any virtual keys 31 of the multiple virtual keys 31 that are different from the first virtual key.

According to the first modified example, when the gesture detection unit 256 detects a second gesture while the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 moves the first virtual key 31 to the target position M1 without moving any virtual keys 31 that are different from the first virtual key among the multiple virtual keys 31. Therefore, if user A wishes to operate only the first virtual key, moving virtual keys 31 that do not need to be moved can be avoided. This reduces the processing required to move virtual keys 31 that do not need to be moved.

Note that in the first modified example, there is a possibility that the first virtual key after movement may overlap with another virtual key 31. If the first virtual key overlaps with another virtual key 31, the operation detection unit 258 may detect an operation on the other virtual key 31, even though user A has operated the first virtual key. For this reason, when the first virtual key overlaps with another virtual key 31, the operation detection unit 258 may disable an operation on the virtual key 31 that overlaps with the first virtual key. In this case, it is possible to prevent the operation detection unit 258 from detecting an operation on the other virtual key 31, even though user A has operated the first virtual key.

B2: Second Modification In the first embodiment and the first modification, the first virtual key is fixed. For example, the virtual key 31H is fixed as the first virtual key.

However, in the first embodiment and the first modified example, the first virtual key may be changed. For example, when the gesture detection unit 256 detects a second gesture when the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 may determine, as the first virtual key, the virtual key that is closest to the target position M1 among the multiple virtual keys 31.

In this case, when the gesture detection unit 256 detects a second gesture while the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 moves the virtual key 31 that is closest to the target position M1 to the target position M1.

For example, if the gesture detection unit 256 detects a second gesture when the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 first obtains the latest virtual space data J5.

Next, the virtual key control unit 257 references the latest virtual space data J5 to determine the virtual key closest to the target position M1 among the multiple virtual keys 31 as the first virtual key.

The virtual key control unit 257 then moves the first virtual key to the target position M1. For example, the virtual key control unit 257 provides a third movement instruction to the virtual key setting unit 254, thereby causing the virtual key setting unit 254 to move the first virtual key to the target position M1. The third movement instruction is an instruction to move the virtual key 31 determined as the first virtual key to the target position M1 by moving the virtual keyboard 3. In response to the third movement instruction, the virtual key setting unit 254 moves the virtual keyboard 3 in the virtual space K2 to move the first virtual key to the target position M1.

The virtual key control unit 257 may provide a fourth movement instruction to the virtual key setting unit 254, thereby causing the virtual key setting unit 254 to move the first virtual key to the target position M1. The fourth movement instruction is an instruction to move the virtual key determined as the first virtual key to the target position M1 without moving any virtual keys 31 of the multiple virtual keys 31 that are different from the virtual key determined as the first virtual key. In response to the fourth movement instruction, the virtual key setting unit 254 moves the virtual key determined as the first virtual key to the target position M1 in the virtual space K2 without moving any virtual keys 31 of the multiple virtual keys 31 that are different from the virtual key determined as the first virtual key.

According to the second modified example, the virtual key control unit 257 determines the virtual key closest to the target position M1 among the multiple virtual keys 31 as the first virtual key. The target position M1 is a position corresponding to the fingertip position of user A's first finger (for example, the middle finger of user A's right hand C). Therefore, user A can change the first virtual key by changing the position of the first finger.

Furthermore, when the gesture detection unit 256 detects a second gesture while the virtual keyboard 3 is located in the virtual space K2, the virtual key control unit 257 moves the virtual key among the multiple virtual keys 31 that is closest to the target position M1 to the target position M1. Therefore, by performing the second gesture with the first finger close to the virtual key to be operated, user A can move the virtual key to be operated to a position that is easy to operate with the first finger.

B3: Third Modification In the first embodiment and the first to second modifications, user A, for example, moves the index finger of his right hand C to position the index finger of his right hand C at the location of the virtual key 31 to be operated.

However, the range of movement of the index finger of right hand C is limited depending on the environment, such as on a train, and user A's posture.

For this reason, in the first embodiment and the first to second variants, the position of the virtual keyboard 3 may be changed based on the movement of, for example, "user A's left hand," which is different from "user A's right hand C."

In this case, the virtual keyboard 3 can be moved by user A's left hand, so the distance that user A's right hand C must move can be shortened compared to a configuration in which user A's left hand cannot move the virtual keyboard 3.

Figure 14 shows the cases where the right hand C moves to the virtual key 31 indicating "A" without moving the virtual keyboard 3, and the case where the right hand C moves to the virtual key 31 indicating "A" by moving the virtual keyboard 3 with the left hand. In Figure 14, movement distance L1 is the movement distance when only the right hand C moves to the virtual key 31 indicating "A" without moving the virtual keyboard 3. Movement distance L2 is the movement distance when the right hand C moves to the virtual key 31 indicating "A" by moving the virtual keyboard 3 with the left hand. As shown in Figure 14, movement distance L2 is shorter than movement distance L1.

Right hand C is an example of the first hand. The first hand is not limited to right hand C, but may be any hand that has a first finger. User A's left hand is an example of the second hand. The second hand is not limited to user A's left hand, but may be any hand different from the first hand.

Figure 15 shows an example of a mobile device 2A according to the third variant. The following explanation focuses on the differences between the mobile device 2A and the mobile device 2 shown in Figure 5.

In addition to the components of portable device 2, portable device 2A includes a motion detection unit 259 and a position change unit 260. The storage device 24 of portable device 2A stores program PG3 instead of program PG2. The processing device 25 of portable device 2A reads program PG3 from the storage device 24. By executing program PG3, the processing device 25 of portable device 2A functions as a hand data generation unit 251, a fingertip detection unit 252, a coordinate conversion unit 253, a virtual key setting unit 254, an image data generation unit 255, a gesture detection unit 256, a virtual key control unit 257, an operation detection unit 258, a motion detection unit 259, and a position change unit 260. At least one of the hand data generation unit 251, fingertip detection unit 252, coordinate conversion unit 253, virtual key setting unit 254, image data generation unit 255, gesture detection unit 256, virtual key control unit 257, operation detection unit 258, movement detection unit 259, and position change unit 260 may be configured using a circuit such as a DSP, ASIC, or FPGA.

The movement detection unit 259 detects the movement of user A's left hand. For example, the movement detection unit 259 detects the movement of user A's left hand based on hand data J1. The movement detection unit 259 detects the movement of the fingers of user A's left hand that perform the second gesture as the movement of user A's left hand. The fingers of user A's left hand that perform the second gesture are, for example, the index finger and middle finger of user A's left hand. The movement of user A's left hand is not limited to the movement of the fingers of user A's left hand that perform the second gesture, but may also be fingers of user A's left hand that are different from the fingers of user A's left hand that perform the second gesture. The movement detection unit 259 detects, as the movement of user A's left hand, the movement direction of the fingers of user A's left hand that perform the second gesture and the movement distance of the fingers of user A's left hand that perform the second gesture. The movement detection unit 259 generates movement detection data J8 based on the detection results of user A's left hand movement. The movement detection data J8 is data that indicates the movement direction of the fingers of user A's left hand when making the second gesture and the movement distance of the fingers of user A's left hand when making the second gesture.

The movement detection unit 259 may detect the movement of user A's left hand in a situation where the coordinate position of user A's left hand in the world coordinate system CS2 is in contact with the virtual keyboard 3. In other words, the movement detection unit 259 may detect the movement of user A's left hand in a situation where user A, who is wearing the MR glasses 1, places part of user A's left hand over the virtual keyboard 3 displayed on the MR glasses 1. In this case, the movement detection unit 259 determines, for example, based on the virtual space data J5 and fingertip data J3, whether the coordinate position of user A's left hand in the world coordinate system CS2 is in contact with the virtual keyboard 3.

The position change unit 260 changes the position of the virtual keyboard 3 based on the movement of user A's left hand detected by the movement detection unit 259. For example, the position change unit 260 moves the virtual keyboard 3 in the virtual space K2 in the direction indicated by the movement detection data J8 by the movement distance indicated by the movement detection data J8. As an example, the position change unit 260 changes the position of the virtual keyboard 3 by providing a keyboard movement instruction to the virtual key setting unit 254. The keyboard movement instruction is an instruction to move the virtual keyboard 3 in the virtual space K2 in the direction indicated by the movement detection data J8 by the movement distance indicated by the movement detection data J8. In response to the keyboard movement instruction, the virtual key setting unit 254 moves the virtual keyboard 3 in the virtual space K2 in the direction indicated by the movement detection data J8 by the movement distance indicated by the movement detection data J8.

The image data generation unit 255 may cause the MR glasses 1 to scroll and display the virtual keyboard 3 in accordance with the movement of the virtual keyboard 3 executed by the position change unit 260. For example, the image data generation unit 255 generates captured image data obtained by capturing an image of the movement of the virtual keyboard 3 executed by the position change unit 260 with the virtual camera N1. The image data generation unit 255 generates image data H1 representing the scrolling of the virtual keyboard 3 by editing the captured image data. The image data generation unit 255 causes the MR glasses 1 to scroll and display the virtual keyboard 3 by transmitting the image data H1 from the communication device 23 to the MR glasses 1. The MR glasses 1 scroll and display the virtual keyboard 3 based on the image data H1. The image data generation unit 255 is an example of a first display control unit.
The MR glasses 1 are an example of a display device.

According to the third modified example, the virtual keyboard 3 can be moved by the left hand of user A. This shortens the distance that user A's right hand C must move compared to a configuration in which the virtual keyboard 3 cannot be moved by user A's left hand.

The second hand used to move the virtual keyboard 3 is the hand (e.g., the left hand) that has the fingers that perform the second gesture. This allows user A to recognize the second hand (e.g., the left hand) that has the fingers that perform the second gesture as the hand used to move the virtual keyboard 3, and the first hand (e.g., the right hand), which is different from the second hand, as the hand that operates the virtual keys 31. This makes it less likely that user A will confuse the hand used to move the virtual keyboard 3 with the hand that operates the virtual keys 31.

The movement detection unit 259 detects the movement of the fingers of user A's left hand performing the second gesture as the second hand movement. As a result, user A can move the virtual keyboard 3 with the fingers performing the second gesture.

When the image data generation unit 255 causes the MR glasses 1 to scroll the virtual keyboard 3 in response to the movement of the virtual keyboard 3 executed by the position change unit 260, the movement of the virtual keyboard 3 can be notified to user A by the scrolling display.

B4: Fourth Modification In the first embodiment and the first to third modifications, the image data generation unit 255 may cause the MR glasses 1 to enlarge and display the virtual key that is closest to the target position M1 among the multiple virtual keys 31.

Figure 16 shows an example of enlarging and displaying the virtual key 31 that is closest to the target position M1 among multiple virtual keys 31. In Figure 16, virtual key 31H is the virtual key that is closest to the target position M1.

For example, the image data generation unit 255 references the latest virtual space data J5 to determine, as the nearby virtual key, the virtual key closest to the target position M1 among the multiple virtual keys 31. The image data generation unit 255 generates captured image data obtained by capturing an image of the virtual space K2 with the virtual camera N1. The image data generation unit 255 edits the captured image data to generate image data H1 representing an image in which only the nearby virtual key is enlarged. The image data generation unit 255 transmits the image data H1 from the communication device 23 to the MR glasses 1, causing the MR glasses 1 to enlarge and display the nearby virtual key. The MR glasses 1 enlarge and display the nearby virtual key based on the image data H1. The image data generation unit 255 is an example of a second display control unit. The MR glasses 1 are an example of a display device.

According to the fourth modification, the image data generation unit 255 causes the MR glasses 1 to enlarge and display the virtual key 31 that is closest to the target position M1 among the multiple virtual keys 31. The virtual key closest to the target position M1 is likely to be the virtual key 31 to be operated. Therefore, the virtual key 31 to be operated can be enlarged and displayed. This makes it easier for user A to recognize the virtual key 31 to be operated compared to a configuration in which the virtual key 31 to be operated is not enlarged and displayed.

B5: Fifth Modification In the first embodiment and the first to fourth modifications, the gesture detection unit 256 may detect the second gesture based on data different from the hand data J1.

Let us assume that the operation button 15 of the MR glasses 1 is located on the body 94 of the MR glasses 1 to accept operation with the fingers of user A's left hand. Furthermore, let us assume that the second gesture is a gesture of pressing the operation button 15. In this case, the gesture detection unit 256 detects the second gesture based on the operation data F1 included in the comprehensive data G1.

According to the fifth variant, the second gesture can be detected without using hand data J1.

B6: Sixth Modification In the first embodiment and the first to fifth modifications, the fingertip detection unit 252 may detect the fingertip position based on the output of a position detection device attached to the fingertip.

According to the sixth variant, the fingertip positions can be detected without using hand data J1.

B7: Seventh Modification Each element realized by the processing device 25 of the mobile device 2 or 2A may be realized by the processing device 18 of the MR glasses 1. In this case, the MR glasses 1 are an example of an information processing device.

According to the seventh variant, the mobile device 2 can be omitted.

C: Others (1) Each function illustrated in Figure 4, Figure 5, or Figure 15 is realized by any combination of hardware and software. There are no particular limitations on how each function is realized. Each function may be realized using a single device that is physically or logically coupled, or may be realized using a device that is configured by connecting two or more physically or logically separated devices directly or indirectly (for example, using wires, wirelessly, etc.). Each function may be realized by combining software with the single device or the multiple devices.

(2) In this specification, the term "apparatus" may be replaced with other terms such as circuit, device, or unit.

(3) In each of the first embodiment and the first to seventh modifications, the storage device 17 and the storage device 24 may be configured with at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. The program may also be transmitted from a network via a telecommunications line.

(4) The first embodiment and the first to seventh variants each include LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (where x is, for example, an integer or a decimal point), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE The present invention may be applied to at least one of systems that use 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), or other appropriate systems, and next-generation systems that are extended, modified, created, or defined based on these. It may also be applied to a combination of multiple systems (for example, a combination of at least one of LTE and LTE-A with 5G).

(5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the first embodiment and each of the first through seventh modifications may be changed as long as there is no contradiction. For example, the methods described herein present various step elements in an exemplary order and are not limited to the specific order presented.

(6) In each of the first embodiment and the first to seventh modifications, input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

(7) In each of the first embodiment and the first to seventh modifications, the determination may be based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of a numerical value (e.g., a comparison with a predetermined value).

(8) The programs exemplified in the first embodiment and each of the first to seventh variations should be broadly construed to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, or functions, regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names. Furthermore, software, instructions, etc. may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber line (DSL)) and wireless technology (such as infrared and microwave), at least one of these wired and wireless technologies is included within the definition of a transmission medium.

(9) The data and other information described in the first embodiment and each of the first to seventh variations may be represented using any of a variety of different technologies. For example, data, information, and other information referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Note that terms described in this specification and terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

(10) In the first embodiment and the first to seventh variations, the terms "system" and "network" are used interchangeably.

(11) In each of the first embodiment and the first to seventh variations, at least one of the MR glasses 1, the mobile device 2, and the mobile device 2A may be a mobile station. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate term.

(12) A mobile station may be referred to as a transmitting device, a receiving device, a communication device, etc. A mobile station may be a device mounted on a mobile object, or the mobile object itself. A mobile object refers to an object that can move. A mobile object can move at any speed. A mobile object can be stopped. Examples of mobile objects include, but are not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcars, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and objects mounted thereon. A mobile object may be an autonomous mobile object that moves based on an operation command. A mobile object may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile object (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). A mobile station also includes devices that do not necessarily move during communication operations. For example, the mobile station may be an IoT (Internet of Things) device such as a sensor.

(13) In each of the first embodiment and the first to seventh variations, the term "determining" may encompass a wide variety of actions. "Determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching a table, database, or other data structure), and regarding ascertaining as having been "determined." Furthermore, "determining" may include regarding receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, or accessing (e.g., accessing data in memory) as having been "judged" or "determined." Furthermore, "decision" can include the act of considering something such as resolving, selecting, choosing, establishing, or comparing to be "decided." In other words, "decision" can include the act of considering some kind of action to be "decided." "Decision" can also be interpreted as "assuming," "expecting," or "considering," among others.

(14) In each of the first embodiment and the first to seventh variations, the term "connected," or any variation thereof, means any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more electric wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

(15) In each of the first embodiment and the first to seventh variations, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

(16) As used herein, any reference to elements using designations such as "first" and "second" does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

(17) When the terms "include," "including," and variations thereof are used in the first embodiment and the first through seventh variations in this specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this specification or claims is not intended to mean an exclusive logical OR.

(18) Throughout this application, where articles are added by translation, such as a, an, and the in English, the disclosure may include the plural form of the noun following these articles.

(19) It will be apparent to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the descriptions in this specification are for illustrative purposes only and do not imply any limitations on the present invention. Furthermore, multiple embodiments selected from the embodiments exemplified in this specification may be combined.

D: Aspects Obtained from the Above-described Embodiments or Modifications The following aspects can be obtained from at least one of the above-described embodiments or modifications.

D1: First Aspect An information processing device according to a first aspect includes a virtual key setting unit, a gesture detection unit, a fingertip detection unit, and a virtual key control unit. The virtual key setting unit positions a first virtual key in a virtual space. The gesture detection unit detects a gesture made by at least one finger. The fingertip detection unit detects the fingertip position of a first finger different from the at least one finger. When the gesture detection unit detects the gesture when the first virtual key is located in the virtual space, the virtual key control unit moves the first virtual key to a target position in the virtual space that corresponds to the fingertip position of the first finger detected by the fingertip detection unit.

This aspect reduces the chance of the virtual key being set in a position other than the correct one.

D2: Second Aspect In the example of the first aspect (second aspect), the first virtual key is one of a plurality of virtual keys included in a virtual keyboard. The virtual key setting unit positions the first virtual key in the virtual space by positioning the virtual keyboard in the virtual space. When the gesture detection unit detects the gesture while the virtual keyboard is positioned in the virtual space, the virtual key control unit moves the first virtual key to the target position. According to this aspect, it is possible to move a virtual key included in the virtual keyboard to a target position.

D3: Third Aspect In the example of the second aspect (third aspect), when the gesture detection unit detects the gesture while the virtual keyboard is located in the virtual space, the virtual key control unit moves the first virtual key to the target position without moving any virtual keys other than the first virtual key among the plurality of virtual keys. According to this aspect, for example, when a user desires to operate only the first virtual key, moving virtual keys that do not need to be moved can be avoided. Therefore, the process of moving virtual keys that do not need to be moved can be reduced.

D4: Fourth Aspect In the example of the second aspect (fourth aspect), when the gesture detection unit detects the gesture in a situation where the virtual keyboard is located in the virtual space, the virtual key control unit moves the first virtual key to the target position by moving the virtual keyboard. According to this aspect, the entire virtual keyboard can be moved in response to the gesture.

D5: Fifth Aspect In any one of the second to fourth aspects (fifth aspect), when the gesture detection unit detects the gesture in a situation where the virtual keyboard is located in the virtual space, the virtual key control unit moves, to the target position, a virtual key among the plurality of virtual keys that is closest to the target position as the first virtual key. According to this aspect, by performing a gesture with a first finger close to a virtual key to be operated, the user can move the virtual key to a position that is easy to operate with the first finger.

D6: Sixth Aspect In any of the examples (sixth aspect) of the second to fifth aspects, the device further includes a movement detection unit that detects the movement of a second hand different from the first hand having the first finger, and a position change unit that changes the position of the virtual keyboard based on the movement of the second hand detected by the movement detection unit. According to this aspect, the virtual keyboard can be moved by the second hand. Therefore, the movement distance of the first hand required to operate a virtual key to be operated can be shortened compared to a configuration in which the virtual keyboard cannot be moved by the second hand.

D7: Seventh Aspect In the example of the sixth aspect (seventh aspect), the device further includes a first display control unit that causes the display device to scroll the virtual keyboard in response to a change in the position of the virtual keyboard executed by the position change unit. According to this aspect, the user can be notified of the movement of the virtual keyboard by the scrolling display.

D8: Eighth Aspect In the example of the sixth or seventh aspect (eighth aspect), the second hand is a hand having at least one finger. According to this aspect, the user can recognize the second hand having the finger making the gesture as the hand related to moving the virtual keyboard, and can recognize the first hand, which is different from the second hand, as the hand operating the virtual keys. Therefore, the user is less likely to confuse the hand related to moving the virtual keyboard with the hand operating the virtual keys.

D9: Ninth Aspect In the example of the eighth aspect (ninth aspect), the movement detection unit detects the movement of the at least one finger as the movement of the second hand. According to this aspect, the user can move the virtual keyboard with the finger making the gesture.

D10: Tenth Aspect In any one of the second to ninth aspects (tenth aspect), the display device further includes a second display control unit that causes the display device to enlarge and display a virtual key among the plurality of virtual keys that is closest to the target position. The virtual key that is closest to the target position is likely to be the virtual key to be operated. Therefore, according to this aspect, the virtual key to be operated can be enlarged and displayed. Therefore, compared to a configuration in which the virtual key to be operated is not enlarged and displayed, the user can more easily recognize the virtual key to be operated.

1...MR glasses, 2...portable device, 2A...portable device, 3...virtual keyboard, 4...image, 11L...camera, 11R...camera, 12...display device, 13...acceleration sensor, 14...gyro sensor, 15...operation button, 16...communication device, 17...storage device, 18...processing device, 19...bus, 21...input device, 22...display device, 23...communication device, 24...storage device, 25...processing device, 26...bus, 31...virtual key, 91...temp 93...bridge, 94...torso, 95...torso, 96L...lens, 96R...lens, 100...information processing system, 181...acquisition unit, 182...motion control unit, 251...hand data generation unit, 252...fingertip detection unit, 253...coordinate conversion unit, 254...virtual key setting unit, 255...image data generation unit, 256...gesture detection unit, 257...virtual key control unit, 258...operation detection unit, 259...motion detection unit, 260...position change unit.

Claims (9)

a virtual key setting unit for positioning a first virtual key in a virtual space;
a gesture detection unit that detects a gesture made by at least one finger;
a fingertip detection unit that detects a fingertip position of a first finger different from the at least one finger;
a virtual key control unit that moves the first virtual key to a target position in the virtual space that corresponds to the fingertip position of the first finger detected by the fingertip detection unit when the gesture detection unit detects the gesture in a situation where the first virtual key is located in the virtual space;
Including,
the first virtual key is one of a plurality of virtual keys included in a virtual keyboard;
the virtual key setting unit positions the first virtual key in the virtual space by positioning the virtual keyboard in the virtual space;
When the gesture detection unit detects the gesture in a situation where the virtual keyboard is located in the virtual space, the virtual key control unit moves the first virtual key to the target position without moving any virtual key different from the first virtual key among the plurality of virtual keys.
Information processing device. a virtual key setting unit for positioning a first virtual key in a virtual space;
a gesture detection unit that detects a gesture made by at least one finger;
a fingertip detection unit that detects a fingertip position of a first finger different from the at least one finger;
a virtual key control unit that moves the first virtual key to a target position in the virtual space that corresponds to the fingertip position of the first finger detected by the fingertip detection unit when the gesture detection unit detects the gesture in a situation where the first virtual key is located in the virtual space;
Including,
the first virtual key is one of a plurality of virtual keys included in a virtual keyboard;
the virtual key setting unit positions the first virtual key in the virtual space by positioning the virtual keyboard in the virtual space;
When the gesture detection unit detects the gesture in a situation where the virtual keyboard is located in the virtual space, the virtual key control unit moves a virtual key among the plurality of virtual keys that is closest to the target position to the target position as a first virtual key.
Information processing device. the virtual key control unit moves the virtual keyboard to move the first virtual key to the target position when the gesture detection unit detects the gesture in a situation where the virtual keyboard is located in the virtual space;
The information processing device according to claim 2 . When the gesture detection unit detects the gesture in a situation where the virtual keyboard is located in the virtual space, the virtual key control unit moves a virtual key among the plurality of virtual keys that is closest to the target position to the target position as a first virtual key.
The information processing device according to claim 1 . a movement detection unit that detects a movement of a second hand different from the first hand having the first finger;
a position change unit that changes a position of the virtual keyboard based on the movement of the second hand detected by the movement detection unit.
The information processing device according to claim 1 . a first display control unit that causes a display device to scroll and display the virtual keyboard in response to a change in the position of the virtual keyboard executed by the position change unit;
The information processing device according to claim 5 . the second hand is a hand having the at least one finger;
7. The information processing device according to claim 5 or 6 . the movement detection unit detects the movement of the at least one finger as the movement of the second hand;
The information processing device according to claim 7 . a second display control unit that causes the display device to enlarge and display a virtual key that is closest to the target position among the plurality of virtual keys;
The information processing device according to claim 1 .