JP5928628B2 - Virtual keyboard input method - Google Patents

Description

  The present invention relates to a method for performing key input using a virtual keyboard displayed on a screen without a physical keyboard in a communication device or information device including a video camera.

  In recent years, mobile devices having a character input function such as a mobile phone with an e-mail function have been widely used. However, the character input method for portable devices that prioritizes small size and light weight specifies a large number of characters with a small number of keys such as dial keys. For example, when inputting “O”, dialing with “A” is assigned. I had to hit the key “1” five times, which was cumbersome. As a technique for improving this, there is a technique related to a virtual keyboard (for example, Patent Document 1).

  The technique described in Patent Document 1 detects the position / motion of an operator's fingertip, generates a hand image based on the motion information of the finger, and creates a virtual keyboard image and hand image having a predetermined key arrangement. The coordinates of the image are superimposed and displayed, and it is determined which key on the virtual keyboard image the fingertip of the hand image image has pressed.

  The means for detecting the position / motion of the operator's fingertip in Patent Document 1 is based on a reference sensor installed at a fixed point and a finger sensor attached to the operator's finger. Specific means include, for example, means for detecting motion by moving image processing of a camera image, measuring finger myoelectric potential, and inputting with various data gloves.

  However, this type of character input method using a virtual keyboard is based on the premise of a blind touch where the user touches the display screen without looking at the key input. It is desirable to have an operability that can easily correct the keystroke mistake.

Japanese Patent No. 4099117

  Accordingly, an object of the present invention is to provide a key input method using a virtual keyboard that can easily correct a keying error when a key near the key to be keyed is accidentally keyed.

In order to solve the above-mentioned problem, the present invention determines which key of an ineffective virtual keyboard has been pressed by a video camera that captures an operation of one or more fingertips of an operator as a video, and inputs the key. A key input method using a virtual keyboard, wherein video data from the video camera is analyzed to detect a fingertip detection step of detecting the position and motion of the one or more fingertips, and information related to the virtual keyboard is displayed. A display step ;
The predetermined specific operation of the fingertip according to a set of home position after detecting, when detecting a substantially stationary state of each of the finger, the position of each of the fingertip becomes the substantially stationary state The virtual keyboard is stored in association with a key at a predetermined home position, and thereafter, each time the fingertip detection step detects any movement of the one or more fingertips, the virtual of the fingertip that detected the movement is stored. calculates a relative moving distance and the moving direction of the keyboard, when the fingertip detection step detects an operation of the fingertip of keying any key constituting the virtual keyboard, the keys keyed by the operation judgment display information according to the key, after displaying the information, the predetermined specific operation of the fingertip which the fingertip detection step according to the cancellation If issued, as well as canceling the input of the key it is determined that the are key depression, and displaying the keys arranged in the vicinity of the key it is determined that the are keying enlarged or emphatically.

  According to the present invention, when a key in the vicinity of a key to be pressed is accidentally keyed, there is an effect that the keying mistake can be easily corrected.

Overall explanatory diagram of the method of the present invention Block configuration diagram of an apparatus to which the method of the present invention is applied Operation flowchart of apparatus to which the method of the present invention is applied

  Hereinafter, a case where the present invention is applied to a key input unit of an information device or a communication device will be described with reference to the drawings.

  FIG. 1 is an overall explanatory view of the method of the present invention, in which 1 is an information device or communication device (hereinafter abbreviated as this device) to which the method of the present invention is applied, 2 is a video camera incorporated in the device 1, 3 is a display unit incorporated in the apparatus 1, 31 is a virtual keyboard image displayed on the display unit 3, 32 is a left-hand image displayed on the display unit 3, 33 is a right-hand image displayed on the display unit 3, 34 is a display image of the input character string, 41 is a virtual keyboard, 42 is the left hand of the operator, and 43 is the right hand of the operator.

  While viewing the virtual keyboard image 31 displayed on the display unit 3, the operator imagines that the virtual keyboard 41 exists on the desk and performs a keystroke operation by blind touch. The left hand 42 and the right hand 43 of the operator who performs the keystroke operation and the fingertips of those hands are photographed by the video camera 2 and subjected to positioning, enlargement / reduction, and rotation as appropriate, and superimposed on the virtual keyboard image 31. And displayed on the display unit 3. The operator knows the position of the key to be pressed from the position of the fingertip of his hand superimposed on the virtual keyboard image 31 displayed on the display unit 3.

  FIG. 2 is a block diagram of the apparatus 1, wherein 11 is a fingertip detection unit, 12 is a specific motion detection unit, 13 is a video composition unit, 14 is a virtual keyboard registration unit, 15 is a key-pressing key determination unit, and 16 is A key input processing unit 17 in the information processing unit 17 is an information processing unit.

  FIG. 3 is an operation flowchart of the apparatus 1. Hereinafter, the operation flow of the apparatus 1 and each functional block will be described with reference to FIGS. The operation flow starts when the power of the apparatus 1 is turned on and the information processing unit 17 waits for key input (S300). In this key input waiting state, the video composition unit 13 reads the virtual keyboard data registered in the virtual keyboard registration unit 14 and displays the virtual keyboard image 31 on the display unit 3.

  In the key input waiting state, the video camera 2 captures the left hand 42 and the right hand 43 of the operator who is performing the keystroke operation, and inputs the video data of the left hand 42 and the right hand 43 of the operator who performs the keystroke operation to the fingertip detection unit 11. (S310).

  The fingertip detection unit 11 analyzes the video data of the left hand 42 and the right hand 43 of the operator who is performing the keystroke operation, which is photographed by the video camera 2, and detects the position of the fingertip and the movement thereof (S311). .

  The specific motion detection unit 12 monitors the motion of each fingertip detected by the fingertip detection unit 11, and detects whether or not there are predetermined specific motions 1, 2, 3,... (S320). The specific action 1 is an action indicating that the fingertips of the left hand 42 and the right hand 43 of the operator are set at the key position of the home position of the virtual keyboard 41. It is an operation of moving back and forth within a certain time in a direction and stopping.

  When the specific motion detection unit 12 detects the specific motion 1 (S320, YES), the position of each fingertip that is stationary after the specific motion 1 is the position of each fingertip held in the key of the home position, and the fingertip detection unit 11 The position of each fingertip is stored. In a general key arrangement, left thumb = “space”, left index finger = “F”, left middle finger = “D”, left ring finger = “S”, left little finger = “A”, right thumb = “space”, right This is the positional relationship between each fingertip and the key where the index finger = “J”, the right middle finger = “K”, the right ring finger = “L”, and the right little finger = “;”.

  The video composition unit 13 then sets the position of each fingertip detected by the fingertip detection unit 11 to the key position of the home position of the virtual keyboard, the virtual keyboard image 31, the left hand image 32, the right hand image 33, or each The position of the fingertip is superimposed and displayed on the display unit 3 (S321), and the process proceeds to S330. If the specific operation 1 is not detected (S320, NO), the process proceeds to S330.

  For the display of the position of each fingertip, the image data of the captured fingertip may be used as it is, or a computer graphics image may be generated and displayed. Alternatively, a mark (for example, “♦”) indicating the position of the fingertip may be displayed.

  Thereafter, the fingertip detection unit 11 continuously analyzes the video data of the left hand 42 and the right hand 43 of the operator who is operating the keystroke, monitors the movement of the fingertip, and detects the movement (YES in S330). Each fingertip updated is calculated by calculating the relative moving distance and moving direction from the home position from the position of each fingertip stored after the specific action 1 and the position of the fingertip after the movement. Is output to the video synthesizing unit 13, and the video synthesizing unit 13 displays the synthesized video by updating the positions of the fingertips of the left hand image 32 and the right hand image 33 displayed on the display unit 3 (S331). ). If no movement is detected (S330, NO), the process proceeds to S340.

  The keying key determination unit 15 monitors the operation of each fingertip detected by the fingertip detection unit 11 and detects the presence or absence of a keying operation (S340). The keystroke operation is a relatively fast reciprocating movement of the finger to be pressed in the vertical direction. When it is detected that this relatively fast reciprocating movement is detected, the finger is moved to the place where the finger was located just before the reciprocating movement. It may be determined that the corresponding key has been pressed.

  It is assumed that the horizontal movement of each finger from the home position is a relatively slow movement. As a result, even if there is only one video camera 2, if each fingertip is photographed obliquely in front, the movement and keystroke operation can be detected, and the motion is horizontal from the two-dimensional moving image of each fingertip. Even if it is difficult to discriminate between the vertical direction and the vertical direction, it is possible to distinguish the horizontal movement of each fingertip and the vertical movement of the keystroke from the difference in movement speed and keystroke speed.

  When the keystroke key determination unit 15 detects a keystroke of any key (S340, YES), the keystroke key determination unit 15 determines the position of the fingertip that performed the keystroke operation and each key registered in the virtual keyboard registration unit 14 The key that has been pressed is determined from the arrangement of the keys, and the key code of the key is transmitted to the information processing unit 17. The information processing unit 17 temporarily stores a key code corresponding to the keystroke key in a temporary memory (not shown) in the key input processing unit 16 and displays the key information on the display unit 3 (S341).

  The character being input displayed on the display unit 3 is a key immediately after the key is pressed or a result of input processing such as character conversion performed by the key input processing unit 16. For example, in the case of the Japanese character input with Roman characters at the end of the character string “ri” in FIG. 1, “ri” is displayed when “r” and “i” are input. The

  When the operator who sees this executes the next character input or key operation to confirm the input character, the keystroke key stored in the temporary memory is confirmed, and the key code or the input processing result is information It is transmitted to the processing unit 17. The information processing unit 17 executes processing by arbitrary application software accompanied by key input.

  On the other hand, if the operator who sees the character being input corresponding to the keystroke key displayed on the display unit 3 notices a keystroke error or a keystroke key determination error, the operator cancels the keystroke. Operation 2 is executed.

  When the specific operation 2 is detected (S350, YES), the process proceeds to S351, where the key pressing key determination unit 15 cancels the key pressing key stored in the temporary memory (backspace key or delete key in a normal keyboard). To the information processing unit 17.

  The information processing unit 17 that has input the key code for canceling the keystroke key clears the key code stored in the temporary memory, and displays the character (for example, the character string 34) displayed on the display unit 3 during the input process. ”“ ”Is deleted. In the case of Japanese input by romaji input, “ri” is displayed when “r” and “i” are input, but specific operation 2 is performed when “r” is input. The keystroke “r” is canceled and “RI” is not displayed.

  Further, in S351, the video composition unit 13 enlarges or highlights a key located in the vicinity of the canceled key in the virtual keyboard image 31 as a candidate for the key to be pressed again (for example, “r” is a keystroke mistake). In the case, “e”, “d”, “f”, “g”, “t”, etc. in the vicinity thereof).

  This is because the key near the target key of the virtual keyboard 41 is often missed, so that it is easy to re-key near or near the key for which the key hit key determination unit 15 has determined key press. Note that any known technique may be applied as a specific means for enlarging or highlighting the key candidate to be pressed again, and detailed description thereof is omitted here.

  Although not shown in detail in the operation flow of FIG. 3, when re-keying due to a keystroke mistake, any key (including other keys in the vicinity) is displayed while a nearby key is enlarged or highlighted. When is pressed again, the enlarged or highlighted display is canceled (not shown).

  The specific operation 2 for canceling the keystroke corresponds to a backspace key or a delete key of a normal keyboard. For example, both hands may be reciprocated left and right as in the specific operation 1. Thereby, it is not necessary to accurately hit the backspace key or the like of the virtual keyboard 41, and it is reliable and simple. Accordingly, it is possible to provide a key input method using a virtual keyboard that can easily correct a keystroke mistake such as erroneously typing adjacent keys.

  Although not shown in the figure, it is possible to freely define the specific operation 3 and later. For example, the specific operation 3 can correspond to a return key, and the specific operation 4 can correspond to a half-width / full-width switching key. However, as the number of specific actions increases, the action of each fingertip for identifying these actions tends to become more complex.

  When the specific operation 2 is not detected (S350, NO), or after the processing of S351, the process proceeds to S360, and it is determined whether or not the key entry key is confirmed. The determination of the key entry key is, for example, determined to be confirmed when there is a key entry of a confirmation key (return key, etc.), and the information processing unit 17 determines the key code stored in the temporary memory (S361), and S370. Proceed to If it is not determined to be confirmed within a certain time, S361 is skipped and the process proceeds to S370. If the key input is not finished (S370, NO), the process returns to S310, and S310 and subsequent steps are repeated.

  If S361 is skipped (S360, NO), the key code of the key whose keystroke is detected in S341 remains stored in the temporary memory, so that it remains in the temporary memory in S341 in the subsequent cycles. If the information processing unit 17 executes the confirmation of the unconfirmed key code, when there is a key input for the next key input as in the normal Japanese character input method, It is possible to confirm (not shown).

  The method for detecting the position and motion of the fingertip by analyzing the video data from the video camera that captured the operator's hands and fingertips has been described. In this method, the specific method for detecting each fingertip of the operator was not mentioned, but for example, using the recognition processing technique, the nail part is extracted and the center position is determined as the fingertip position. do it.

  Further, as a method for determining the position of the fingertip by a simple process, there is a method of attaching an object having a feature point such as nail art to each nail or drawing a picture or mark having a feature point. Alternatively, there is a method of wearing a glove or sack having a characteristic point on the fingertip.

  By the way, in the embodiment of the present invention, as a method of detecting the position and movement of the fingertip, the method of detecting video data from the video camera 2 by image processing has been described, but instead of performing video processing of video data, A method by direct sensor input is also applicable. For example, a magnetic material or a wireless tag may be attached to the nail or glove of each fingertip, and their position and operation may be detected by a predetermined sensor (for example, a hall element or a wireless tag reader).

DESCRIPTION OF SYMBOLS 1 ... Apparatus to which the method of the present invention is applied 2 ... Video camera 3 ... Display unit 11 ... Fingertip detection unit 12 ... Specific motion detection unit 13 ... Video composition unit 14 ... Virtual Keyboard registration unit 15 ... keystroke key determination unit 16 ... key input processing unit 17 ... information processing unit 31 ... virtual keyboard image 32 ... left-hand image 33 ... right-hand image 34 ... input Character string 41 ... virtual keyboard 42 ... left hand 43 ... right hand

Claims (1)

A key input method using a virtual keyboard that determines which key of an ineffective virtual keyboard has been pressed by a video camera that captures an operation of one or more fingertips of an operator as a video,
Analyzing the video data from the video camera, and detecting a fingertip position and movement of the one or more fingertips; and a display step of displaying information related to the virtual keyboard ,
The predetermined specific operation of the fingertip according to a set of home position after detecting, when detecting a substantially stationary state of each of the finger, the position of each of the fingertip becomes the substantially stationary state Store it in association with the key of the predetermined home position on the virtual keyboard,
Thereafter, each time the fingertip detection step detects any movement of the one or more fingertips, the relative movement distance and movement direction of the detected fingertip within the virtual keyboard is calculated, and the fingertip detection is performed. If the step has detected the operation of the fingertip of keying any key constituting the virtual keyboard, display information relating to the determined corresponding key key keyed by the operation,
After displaying the information, when the fingertip detection step detects a predetermined specific action of the fingertip related to cancellation, the input of the key determined to be pressed is canceled and it is determined that the key has been pressed. A virtual keyboard input method, wherein a key arranged in the vicinity of a key is displayed in an enlarged or highlighted manner.

Images