JP7670565B2 - Input device, input method, and input program - Google Patents

Description

Patent Act Article 30, Paragraph 2 applied 1. Retail Tech JAPAN 2021 Announced on March 9, 2021 2. Sharp Document Circle Announced on April 22, 2021

The present invention relates to an input device, an input method, and an input program that detects a gesture operation by a user's hand and performs input processing at an input position on a display screen.

Conventionally, input devices that allow input (screen operation) by gesture operations on the display screen of a display panel are known. For example, Patent Document 1 discloses a device that determines input information by grasping at least one of the shape, movement, and position of a user's hand from an image captured of the hand in a virtual space.

JP 2000-75991 A

Here, the input device identifies a position on the display screen (input position) designated by, for example, a gesture operation using the user's hand, based on the position of the hand. The user also performs input operations by moving their hand or changing the shape of their hand while looking at a mouse cursor image displayed on the display screen. However, with conventional technology, a problem occurs in that it is difficult for the user to grasp the correspondence between the range in which the user moves their hand and the movable range within the display screen of the mouse cursor image that moves on the display screen in response to the movement of the user's hand. For example, it is difficult for the user to grasp how far the mouse cursor image on the display screen will move when the user moves their hand a specified distance. Thus, conventional input devices have a problem of low operability for users.

The object of the present invention is to provide an input device, an input method, and an input program that can improve the operability of an input device that detects a user's gesture operation and executes input processing at an input position on a display screen.

An input device according to one aspect of the present invention is a device that includes an acquisition processing unit that acquires an image of a user, a detection processing unit that detects the user's hand from the captured image acquired by the acquisition processing unit, a first setting processing unit that sets an operation area for the gesture operation that corresponds to an inputtable area on the display screen that can be input by the gesture operation and that includes the user's hand detected by the detection processing unit, and a display processing unit that displays the captured image acquired by the acquisition processing unit on the display screen and displays an operation area image corresponding to the operation area set by the first setting processing unit superimposed on the captured image.

An input method according to another aspect of the present invention is an input method for detecting a gesture operation by a user's hand and executing input processing at an input position on a display screen, in which one or more processors execute the following steps: an acquisition step of acquiring an image of the user; a detection step of detecting the user's hand from the image acquired in the acquisition step; a first setting step of setting an operation area for the gesture operation that corresponds to an inputtable area on the display screen that includes the user's hand detected in the detection step and that can be input by the gesture operation; and a display step of displaying the image acquired in the acquisition step on the display screen and displaying an operation area image corresponding to the operation area set in the first setting step superimposed on the image.

An input program according to another aspect of the present invention is an input program that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen, the input program causing one or more processors to execute the following steps: an acquisition step of acquiring an image of the user; a detection step of detecting the user's hand from the image acquired in the acquisition step; a first setting step of setting an operation area for the gesture operation that corresponds to an inputtable area on the display screen that includes the user's hand detected in the detection step and that can be input by the gesture operation; and a display step of displaying the image acquired in the acquisition step on the display screen and displaying an operation area image corresponding to the operation area set in the first setting step superimposed on the image.

The present invention provides an input device, an input method, and an input program that can improve the operability of an input device that detects a user's gesture operation and executes input processing at an input position on a display screen.

FIG. 1 is a block diagram showing a configuration of a display device according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing an example of a display screen 13A in the display device according to the embodiment of the invention. FIG. 3 is a diagram showing an example of a captured image displayed on the display device according to the embodiment of the present invention. FIG. 4 is a diagram showing a method of setting a reference point by the first setting method according to the embodiment of the present invention. FIG. 5 is a diagram showing an example of a captured image displayed on the display device according to the embodiment of the present invention. FIG. 6 is a diagram showing a method of setting a reference point by the first setting method according to the embodiment of the present invention. FIG. 7 is a diagram showing a method of setting a reference point by the second setting method according to the embodiment of the present invention. FIG. 8 is a diagram showing a method of setting a reference point by the third setting method according to the embodiment of the present invention. FIG. 9 is a diagram showing an example of a rectangular area in a captured image displayed on the display device according to the embodiment of the present invention. FIG. 10 is a diagram showing an example of an operation area in a captured image displayed on the display device according to the embodiment of the present invention. FIG. 11 is a diagram showing an example of an operation area in a captured image displayed on the display device according to the embodiment of the present invention. FIG. 12 is a diagram showing a method of calculating an input position by the first calculation method according to the embodiment of the present invention. FIG. 13 is a diagram showing a method of calculating an input position by the second calculation method according to the embodiment of the present invention. FIG. 14 is a diagram showing a method of calculating an input position by the third calculation method according to the embodiment of the present invention. FIG. 15 is a flowchart illustrating an example of a procedure of a display control process executed by the display device according to the embodiment of the present invention. FIG. 16 is a flowchart illustrating an example of a procedure of a reference point setting process by the first setting method according to an embodiment of the present invention. FIG. 17 is a flowchart illustrating an example of a procedure of a reference point setting process using the second setting method according to an embodiment of the present invention. FIG. 18 is a flowchart illustrating an example of a reference point setting process procedure according to the third setting method according to the embodiment of the present invention. FIG. 19 is a flowchart illustrating an example of a procedure of an input position calculation process using the first calculation method according to an embodiment of the present invention. FIG. 20 is a flowchart illustrating an example of a procedure of an input position calculation process using the second calculation method according to an embodiment of the present invention. FIG. 21 is a flowchart illustrating an example of a procedure of an input position calculation process using the third calculation method according to an embodiment of the present invention.

Hereinafter, an embodiment of the present invention will be described with reference to the attached drawings. Note that the following embodiment is an example of the present invention and does not limit the technical scope of the present invention.

As shown in FIG. 1, a display device 1 according to an embodiment of the present invention includes a control unit 11, a memory unit 12, a display panel 13, an operation unit 14, and a camera 15. FIG. 2 shows a schematic diagram of the display device 1. The camera 15 is installed on the top of the display panel 13 and captures an image of a predetermined area in front of the display panel 13. The display device 1 is an example of an input device of the present invention. The input device of the present invention is not limited to a display device, and may be an information processing device such as a server capable of data communication with the display panel 13 and the camera 15.

The display device 1 accepts a non-contact input operation by the user on the display screen 13A. For example, when the display device 1 detects an input operation (gesture operation) by the user's gesture in front of the camera 15, it executes input processing according to the user's input operation on the display screen 13A. For example, when the user closes his/her hand (making a fist) in front of the camera 15, the display device 1 accepts a touch input at the position of the mouse cursor M1 on the display screen 13A. The specific configuration of the display device 1 will be described below.

Camera 15 is a digital camera that captures an image of a subject and outputs it as digital image data. Camera 15 captures images at a predetermined frame rate and sequentially outputs the digital image data to control unit 11. There may be one camera 15 or multiple cameras 15. Specifically, camera 15 is installed above display panel 13 and captures an image of a predetermined area in front of display panel 13.

The display panel 13 is a display that displays images on the display screen 13A, and is, for example, a liquid crystal display. The operation unit 14 is an operation device such as a mouse and a keyboard. The operation unit 14 may be configured as a touch panel.

The display panel 13 displays various images according to instructions from the control unit 11. For example, as shown in FIG. 2, the display panel 13 displays a mouse cursor M1 (an example of an input operation icon of the present invention) that moves following the user's hand, operation buttons (here, "Yes" button and "No" button) that accept touch input in response to a predetermined gesture operation by the user's hand, and a captured image Wp (wipe screen) captured by the camera 15. FIG. 3 shows an enlarged view of the captured image Wp. Although details will be described later, the captured image Wp displays a frame image representing the outer frame of a rectangular area A1 (an example of a specific area of the present invention) surrounding the user's hand, and a frame image representing the outer frame of an operation area A11 corresponding to an inputtable area F1 (see FIG. 2) on the display screen 13A that can be input. The user moves the position of the mouse cursor M1 or selects (clicks) an operation button with the mouse cursor M1 while looking at the captured image Wp displayed superimposed on the display screen 13A. For example, the user moves the display position of the mouse cursor M1 by moving the mouse cursor M1 with an open hand (open hand) (see FIG. 3), and when the mouse cursor M1 is located over the "Yes" button, the user selects (clicks) the "Yes" button by closing the hand (open hand) (see FIG. 5). The operation of opening the hand is an example of the first gesture operation of the present invention, and the operation of closing the hand is an example of the second gesture operation of the present invention.

The storage unit 12 is a non-volatile storage unit such as a HDD (Hard Disk Drive) or SSD (Solid State Drive) that stores various information. The storage unit 12 stores control programs such as a display control program for causing the control unit 11 to execute a display control process (see FIG. 15, etc.) described below. For example, the display control program is non-temporarily recorded on a computer-readable recording medium such as a CD or DVD, and is read by a reading device (not shown) such as a CD drive or DVD drive provided in the display device 1 and stored in the storage unit 12. The display control program may be distributed from a cloud server and stored in the storage unit 12.

The control unit 11 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic processes. The ROM is a non-volatile storage unit in which control programs such as a BIOS and an OS for causing the CPU to execute various arithmetic processes are stored in advance. The RAM is a volatile or non-volatile storage unit that stores various information, and is used as a temporary storage memory (work area) for various processes executed by the CPU. The control unit 11 controls the display device 1 by having the CPU execute various control programs that are stored in advance in the ROM or the storage unit 12.

However, in conventional technology, for example, when a user changes from opening his/her hand (see FIG. 3) to closing his/her hand (see FIG. 5), the input position on the display screen 13A changes in a direction that is not intended by the user. A specific example of this problem is described below.

4 shows an enlarged view of an open hand, and FIG. 6 shows an enlarged view of a closed hand. In conventional technology, the center position P0 of the entire area (rectangular area A1) onto which the hand is projected during gesture operation is specified as the input position, so when the hand is changed from an open state to a closed state, the center position P0 moves to the center position P0' of the rectangular area A2 as shown in FIG. 6. Therefore, when the user changes the hand from an open state to a closed state, the position of the mouse cursor M1 corresponding to the center position P0 also moves, making it impossible for the user to correctly perform the intended operation, for example, the selection operation of the "Yes" button. In contrast, the display device 1 according to this embodiment can prevent the input position from changing in an unintended direction due to a change in the user's gesture operation, as shown below.

Specifically, as shown in FIG. 1, the control unit 11 includes various processing units such as a display processing unit 111, an acquisition processing unit 112, a detection processing unit 113, a judgment processing unit 114, a reference point setting processing unit 115, an operation area setting processing unit 116, a virtual inputtable area setting processing unit 117, and an input processing unit 118. The control unit 11 functions as each processing unit by executing various processes according to the display control program and by the CPU. Some or all of the processing units included in the control unit 11 may be configured with electronic circuits. The display control program may be a program for causing multiple processors to function as the various processing units.

The display processing unit 111 displays various information on the display panel 13. The display processing unit 111 displays various images according to the execution of various applications. For example, the display processing unit 111 displays the display screen 13A shown in FIG. 2 on the display panel 13. The display processing unit 111 also displays the captured image Wp on the end (corner) of the display screen 13A. The captured image Wp is a partial image of a specific range including a detected user's face image among the captured images of a predetermined area captured by the camera 15. The display processing unit 111 displays the captured image Wp on a small screen in the corner of the display screen 13A. The display processing unit 111 also updates the captured image Wp in real time according to the captured image acquired from the camera 15. The display processing unit 111 also displays the captured image Wp on the display screen 13A, and displays a frame image (an example of an operation area image of the present invention) corresponding to the operation area A11 superimposed on the captured image Wp. The display position of the captured image Wp is not limited. The display processing unit 111 may also display the captured image Wp on the condition that the user's face or hand is detected.

The display processing unit 111 also displays a mouse cursor M1 (icon for input operation) at an input position corresponding to the reference point P1 within an input area F1 that is set a predetermined width inward from the bottom edge of the display screen 13A. The predetermined width is set to the vertical width of the captured image Wp, for example. The user can move the mouse cursor M1 within the input area F1. The display processing unit 111 is an example of the display processing unit of the present invention.

The acquisition processing unit 112 acquires captured images of the user. Specifically, the acquisition processing unit 112 sequentially acquires digital image data of images captured by the camera 15 at a predetermined frame rate from the camera 15. The acquisition processing unit 112 is an example of the acquisition processing unit of the present invention.

The detection processing unit 113 detects a user (e.g., a face) from the captured image acquired by the acquisition processing unit 112. The detection processing unit 113 also detects the user's hands from the captured image. Well-known methods can be applied as a method for detecting the user's face and hands. For example, the detection processing unit 113 performs processing such as cropping, resizing, flipping, and rotating on the digital image data to detect the user's face and hands. When the detection processing unit 113 detects a hand, it tracks the movement of the hand. The detection processing unit 113 is an example of a detection processing unit of the present invention.

The determination processing unit 114 determines the shape of the user's hand detected by the detection processing unit 113. For example, the determination processing unit 114 determines whether the user's hand is in an open state (paper state) (see FIG. 3) or a closed state (rock state) (see FIG. 5). The determination processing unit 114 also determines whether the user's hand detected by the detection processing unit 113 is a right hand or a left hand. The determination processing unit 114 is an example of a determination processing unit of the present invention.

The reference point setting processing unit 115 sets a reference point P1 at a predetermined position on the bottom side of the rectangular area A1 that surrounds the user's hand detected by the detection processing unit 113. Specifically, the reference point setting processing unit 115 sets the reference point P1 to a position obtained by dividing the distance from one end E1 to the other end E2 on the bottom side of the rectangular area A1 at a predetermined ratio. The reference point setting processing unit 115 is an example of the reference point setting processing unit of the present invention.

The reference point P1 is a position in the virtual space corresponding to the input position on the display screen 13A. The input processing unit 118 executes input processing for the input position corresponding to the reference point P1. Specifically, when the shape of the user's hand detected by the detection processing unit 113 is a predetermined shape, the input processing unit 118 executes input processing for the input position corresponding to the reference point P1 set by the reference point setting processing unit 115. For example, the input processing unit 118 displays a mouse cursor M1 on the display screen 13A at a position where the reference point P1 is mapped to an area corresponding to the display screen 13A. The input processing unit 118 is an example of an input processing unit of the present invention.

Specific methods for setting the reference point P1 are described below. Here, the first setting method, the second setting method, and the third setting method are given as specific examples of the setting method.

[First setting method]
FIG. 4 shows an example of the first setting method. As shown in FIG. 4, the reference point setting processing unit 115 sets the center position of the lower side L1 of the rectangular area A1 to the reference point P1. Here, the rectangular area A1 is a rectangular area that contacts the outermost end of the hand in an open state, and the reference point P1 is the center position of the lower side L1 of the rectangular area A1 in an open state. Therefore, as shown in FIG. 6, even if the hand is closed, the position of the reference point P1 does not change. In this way, the reference point setting processing unit 115 sets the center position of the lower side L1 of the rectangular area A1 in an open state to the reference point P1 regardless of the change in the shape of the hand. Note that the reference point setting processing unit 115 may set the center position of the lower side L1 of the rectangular area A1 in a closed state to the reference point P1.

In the first setting method, the input processing unit 118 calculates the coordinates of the center position of the bottom side L1 of the rectangular area A1 as the coordinates of the input position. For example, if the width of the rectangular area A1 is W1, the height is H1, and the upper left end E0 is the origin [x, y], the input processing unit 118 calculates the coordinates of the reference point P1 as [x + (W1)/2, y + H1]. The input processing unit 118 calculates the position coordinates of the mouse cursor M1 based on the calculated coordinates of the reference point P1. The display processing unit 111 displays the mouse cursor M1 at the calculated position coordinates on the display screen 13A.

[Second setting method]
FIG. 7 shows an example of the second setting method. As shown in FIG. 7, the reference point setting processing unit 115 sets a position on the bottom side L1 of the rectangular area A1 at a ratio of 1:2 from the end of the little finger side of the user's hand as the reference point P1. Here, when the hand is changed from an open state (see FIG. 4) to a closed state (see FIG. 6), the horizontal position of the center position P0 moves toward the little finger side. This movement amount is easily affected by the area and movement amount of the thumb. Therefore, the reference point setting processing unit 115 assumes that the 1/3 area from the thumb side of the rectangular area A1 is the thumb area, extracts the palm area excluding the thumb area (2/3 area on the little finger side of the rectangular area A1), and sets the center position of the bottom side L1 of the palm area to the reference point P1. That is, the reference point setting processing unit 115 sets a position on the bottom side L1 of the rectangular area A1 at a ratio of 1:2 from the end E1 of the little finger side of the user's hand as the reference point P1.

In the second setting method, the input processing unit 118 calculates the coordinates of the input position at a ratio of 1:2 from the end E1 of the bottom side L1 of the rectangular area A1 on the little finger side of the user's hand. For example, the input processing unit 118 calculates [x+(W1)/3, y+H1] as the coordinates of the reference point P1. The input processing unit 118 calculates the position coordinates of the mouse cursor M1 based on the calculated coordinates of the reference point P1. The display processing unit 111 displays the mouse cursor M1 at the calculated position coordinates on the display screen 13A.

[Third setting method]
FIG. 8 shows an example of the third setting method. As shown in FIG. 8, the reference point setting processing unit 115 specifies a specific pixel that displays the color of the user's hand among the multiple pixels included in the rectangular area A1, calculates the total number of the specific pixels for each pixel row in the column direction from the upper side to the lower side L1 of the rectangular area A1, and sets the center position Xc of the partial side corresponding to the range of the pixel row in the lower side L1 where the total number of pixels exceeds the threshold value Cth as the reference point P1. The graph shown in FIG. 8 shows the total number of the specific pixels for each pixel row. The reference point setting processing unit 115 calculates the minimum position Xmin and the maximum position Xmax of the pixel row in which the total number of pixels exceeds the threshold value Cth, and calculates the center position Xc of the partial side from the minimum position Xmin to the maximum position Xmax. Then, the reference point setting processing unit 115 sets the calculated center position Xc as the reference point P1.

In the third setting method, the input processing unit 118 calculates the coordinates of the center position Xc of the partial side of the lower side L1 that corresponds to the range of the pixel row in which the total pixel number exceeds the threshold value Cth as the coordinates of the input position. For example, if the coordinates of the minimum position are [Xmin, y+H1], the input processing unit 118 calculates the coordinates of the reference point P1 as [Xmin+(Xmax-Xmin)/2, y+H1]. The input processing unit 118 calculates the position coordinates of the mouse cursor M1 based on the calculated coordinates of the reference point P1. The display processing unit 111 displays the mouse cursor M1 at the calculated position coordinates on the display screen 13A.

The reference point setting processing unit 115 sets the reference point P1 using one of the first setting method, the second setting method, and the third setting method described above.

According to each of the above setting methods, by setting the reference point P1 on the bottom side L1 of the rectangular area A1, it is possible to prevent misalignment in the vertical direction. In addition, by using information on the rectangular area A1 in the shape of an open hand (first gesture operation), it is possible to prevent misalignment of the mouse cursor M1 even when the shape changes to a rectangular area A2 (see FIG. 6) in the shape of a closed hand (second gesture operation).

In addition, according to the first setting method, the reference point P1 can be fixed to the center of the lower side L1, which reduces the amount of calculation required for the input position. Therefore, this method is effective when the CPU has low processing power or when high-speed processing is required.

In addition, according to the second setting method, by assuming that the thumb area is approximately 1/3 of the rectangular area A1 of the hand, the reference point P1 can be set by focusing on the palm area, which has little change in area. Therefore, the horizontal input position can be calculated with high accuracy and stability.

In addition, according to the third setting method, the reference point P1 can be set narrowed down to the palm area by excluding the thumb area. This allows the lateral input position to be calculated more accurately and stably.

The input processing unit 118 executes input processing for an input position corresponding to the reference point P1 set by the reference point setting processing unit 115 using any of the setting methods described above. Specifically, when the shape of the user's hand detected by the detection processing unit 113 is an open hand shape (first shape), the input processing unit 118 executes a movement process to move the mouse cursor M1 displayed on the display screen 13A in accordance with the movement of the reference point P1. In addition, when the shape of the user's hand detected by the detection processing unit 113 changes from an open hand shape to a closed hand shape (second shape), the input processing unit 118 executes a selection process for a selection target (for example, the "Yes" button or the "No" button in FIG. 2) at the display position of the mouse cursor M1 corresponding to the reference point P1.

Here, the display device 1 identifies a position (input position) on the display screen 13A designated by, for example, a gesture operation with the user's hand based on the position of the hand. The user performs an input operation by moving his/her hand or changing the shape of his/her hand while looking at the mouse cursor M1 displayed on the display screen 13A. However, with the conventional technology, a problem occurs in that it is difficult for the user to grasp the correspondence between the range in which the user moves his/her hand and the movable range within the display screen 13A of the mouse cursor M1 that moves on the display screen 13A in response to the movement of the user's hand. For example, it is difficult for the user to grasp how far the mouse cursor M1 on the display screen 13A will move when the user moves his/her hand by a predetermined distance. Thus, with the conventional technology, there is a problem of low operability for the user. In contrast, with the display device 1 according to this embodiment, it is possible to improve the operability of the display device 1 as shown below.

Specifically, the operation area setting processing unit 116 sets an operation area A11 for gesture operation that corresponds to an inputtable area F1 (see FIG. 2) that can be input by a gesture operation in the display screen 13A, which is an area including the user's hand detected by the detection processing unit 113. Specifically, when the detection processing unit 113 detects the user's hand, the operation area setting processing unit 116 sets a rectangular area A1 that surrounds the outline of the hand (see FIG. 9), and sets an operation area A11 that is larger than the rectangular area A1 using a preset margin (see FIG. 10). For example, the operation area setting processing unit 116 sets the operation area A11 at the position of the preset upper, lower, left, and right margins m1 to m4 (see FIG. 10) in a partial image (W×3/5 area) that includes the rectangular area A1 of the detected hand in the captured image Wp. In addition, the operation area setting processing unit 116 determines whether the hand is a left hand or a right hand, and determines the direction (left or right) in which the operation area A11 is set. The margins m1 to m4 are set to values that allow the mouse cursor M1 to move left/right and up/down over the entire input area F1 (see FIG. 2) when the user moves their hand left/right and up/down over the entire area of the operation area A11.

For example, the operation area setting processing unit 116 may set the operation area A11 to an area obtained by adding half the length (W1)/2 of the width W1 of the rectangular area A1 to the little finger side of the rectangular area A1, adding half the length (H1)/2 of the height H1 of the rectangular area A1 to the upper side of the rectangular area A1, and adding the length of the height H1 of the rectangular area A1 to the lower side of the rectangular area A1. The operation area setting processing unit 116 is an example of a first setting processing unit of the present invention.

Here, since the position of the bottom side of the rectangular area A1 of the hand (reference point P1) corresponds to the position of the mouse cursor M1, if the operation area A11 is set based on the reference point P1, when the user moves his/her hand upward, the mouse cursor M1 cannot be moved to the top of the display screen 13A unless the hand is moved to a state where the hand protrudes above the operation area A11, which gives the user a sense of discomfort. Therefore, by displaying the operation area A11 above its actual position, the user can move the mouse cursor M1 within the operation area A11 without feeling uncomfortable. Specifically, the display processing unit 111 displays the captured image Wp on the display screen 13A, and also displays a frame image representing the outer frame of the operation area A11 shifted above its actual position with respect to the captured image Wp. For example, as shown in FIG. 11, the display processing unit 111 displays a frame image representing the outer frame of the operation area A11 on the display screen 13A, shifted upward from the actual position (dotted frame in FIG. 11) by half the width h ((H1)/2) of the rectangular area A1 in the vertical direction with respect to the captured image Wp. This allows the user to move the mouse cursor M1 in accordance with the movement of his or her hand, with the sensation that the center position of the palm corresponds to the position of the mouse cursor M1.

The virtual inputable area setting processing unit 117 sets a virtual inputable area F2 (virtual screen) larger than the inputable area F1. For example, as shown in FIG. 12, if the width of the display screen 13A is W2 and the height of the display screen 13A is H2, the virtual inputable area setting processing unit 117 adds half the length ((W2)/2) of the width W2 of the display screen 13A to each of the right and left sides of the display screen 13A, and adds half the length ((H2)/2) of the height H2 of the display screen 13A to each of the upper and lower sides of the display screen 13A, and sets the virtual inputable area F2 to an area (an area twice the length and width of the inputable area F1). The virtual inputable area setting processing unit 117 is an example of a second setting processing unit of the present invention.

The input processing unit 118 uses the operation area A11, the inputtable area F1, and the virtual inputtable area F2 to calculate the input position on the display screen 13A and perform input processing.

Specific methods for calculating the input position are described below. Here, the first calculation method, the second calculation method, and the third calculation method are given as specific examples of the calculation method.

[First calculation method]
FIG. 12 shows an example of the first calculation method. As shown in FIG. 12, the display processing unit 111 displays a frame image of the outer frame of the operation area A11 in the captured image Wp. The input processing unit 118 also calculates a position in the operation area A11 corresponding to the reference point P1 in the virtual input possible area F2 as the input position. Specifically, the input processing unit 118 maps the reference point P1 of the operation area A11 to the virtual input possible area F2. Then, when the mapped position (the position of the mouse cursor M1) is within the input possible area F1, the input processing unit 118 determines the position as the input position and displays the mouse cursor M1 at the input position. On the other hand, when the mapped position (the position of the mouse cursor M1) protrudes outside the input possible area F1, the input processing unit 118 remaps the position within the input possible area F1, determines the position of the end of the input possible area F1 as the input position, and displays the mouse cursor M1 at the input position.

In this way, in the first calculation method, when the input processing unit 118 maps the operation area A11 to the inputtable area F1, it maps the operation area A11 to a virtual inputtable area F2 that is twice the vertical and horizontal size of the actual screen size of the display screen 13A, and if the mouse cursor M1 is mapped outside the inputtable area F1, it is remapped into the inputtable area F1.

[Second calculation method]
FIG. 13 shows an example of the second calculation method. As shown in FIG. 13, the display processing unit 111 displays a frame image of the outer frame of the virtual operation area B1, which is larger than the operation area A11, in the captured image Wp. For example, the display processing unit 111 displays a frame image of an extended area (virtual operation area B1) in which half the width of the operation area A11 is added to the right and left sides of the operation area A11, and half the length of the vertical width of the operation area A11 is added to the upper and lower sides of the operation area A11, in the captured image Wp. That is, the display processing unit 111 does not display a frame image of the operation area A11 in the captured image Wp, but displays a frame image of a virtual operation area B1 that is a virtual extension of the operation area A11. This allows the user to move his/her hand while looking at the frame image of the virtual operation area B1 displayed in the captured image Wp.

The input processing unit 118 calculates the position in the operation area A11 corresponding to the reference point P1 in the input area F1 as the input position. Specifically, the input processing unit 118 maps the reference point P1 of the operation area A11 to the input area F1. If the mapped position (the position of the mouse cursor M1) is within the input area F1, the input processing unit 118 determines that position as the input position and displays the mouse cursor M1 at that input position. On the other hand, if the mapped position (the position of the mouse cursor M1) is outside the input area F1, the input processing unit 118 remaps it within the input area F1, determines the position of the edge of the input area F1 as the input position, and displays the mouse cursor M1 at that input position.

In this way, in the second calculation method, a frame image of the virtual operation area B1, which is twice as large vertically and horizontally as the operation area A11, is displayed in the captured image Wp, while the input processing unit 118 maps the operation area A11 to the inputtable area F1, and if the mouse cursor M1 is mapped outside the inputtable area F1, it is remapped into the inputtable area F1.

[Third calculation method]
FIG. 14 shows an example of the third calculation method. As shown in FIG. 14, the display processing unit 111 displays a frame image of the virtual operation area B1, which is larger than the operation area A11, in the captured image Wp, in the same manner as the second calculation method. That is, the display processing unit 111 does not display a frame image of the operation area A11 in the captured image Wp, but displays a frame image of the virtual operation area B1. The input processing unit 118 also calculates a position in the virtual operation area B1 corresponding to the reference point P1 in the virtual input possible area F2 as the input position. Specifically, the input processing unit 118 maps the reference point P1 of the virtual operation area B1 to the virtual input possible area F2. Then, if the mapped position (the position of the mouse cursor M1) is within the input possible area F1, the input processing unit 118 determines the position as the input position and displays the mouse cursor M1 at the input position. On the other hand, if the mapped position (the position of the mouse cursor M1) falls outside the input area F1, the input processing unit 118 remaps it within the input area F1, determines the position of the edge of the input area F1 as the input position, and displays the mouse cursor M1 at that input position.

In this way, in the third calculation method, a frame image of a virtual operation area B1 that is twice as large vertically and horizontally as the operation area A11 is displayed in the captured image Wp, while the input processing unit 118 maps the virtual operation area B1 to the virtual inputtable area F2, and if the mouse cursor M1 is mapped outside the inputtable area F1, it is remapped into the inputtable area F1.

The input processing unit 118 calculates the input position using one of the first, second, and third calculation methods described above.

According to the first calculation method, the operation area A11 is made relatively narrow compared to the input area F1, so the user can move the mouse cursor M1 with small hand movements, and can move the mouse cursor M1 over the entire display screen 13A by moving the hand within the small area.

In addition, according to the second calculation method, the operation area A11 can be made wider than the input area F1, so the user can reflect finer hand movements in the movement of the mouse cursor M1 than with the first calculation method.

In addition, according to the third calculation method, the operation area A11 can be set to an appropriate size relative to the input area F1, so that finer hand movements can be reflected in the movement of the mouse cursor M1 than with the first calculation method, and the mouse cursor M1 can be moved over the entire display screen 13A without moving the hand or arm as much as with the second calculation method.

In addition, according to each of the calculation methods described above, it is possible to prevent the mouse cursor M1 from going off the display screen 13A or from moving into the captured image Wp (wipe screen) below the display screen 13A, which would cause a malfunction.

As described above, when the display device 1 detects the user's hand from the image captured by the camera 15, it sets a reference point P1 on the bottom side L1 of the rectangular area A1 of the hand using a predetermined setting method (the first to third setting methods), calculates the input position of the reference point P1 in the input area F1 using a predetermined calculation method (the first to third calculation methods), and performs input processing in accordance with the user's gesture operation at the calculated input position.

[Display control process]
Hereinafter, the display control process executed by the control unit 11 of the display device 1 will be described with reference to FIG.

The present invention can be understood as an invention of a display control method (one example of an input method of the present invention) that executes one or more steps included in the display control process, and one or more steps included in the display control process described here may be omitted as appropriate. The steps in the display control process may be executed in a different order as long as the same action and effect is achieved. Furthermore, although an example is described here in which the control unit 11 executes each step in the display control process, another embodiment can also be a display control method in which multiple processors execute each step in the display control process in a distributed manner.

First, in step S1, the control unit 11 determines whether or not a captured image has been acquired from the camera 15. When the control unit 11 acquires a captured image from the camera 15 (S1: Yes), the process proceeds to step S2.

In step S2, the control unit 11 determines whether or not a user has been detected. If the control unit 11 detects a user (S2: Yes), the process proceeds to step S3.

In step S3, the control unit 11 determines whether the user's face and hands have been detected. If the control unit 11 detects the user's face and hands (S3: Yes), the process proceeds to step S4.

In step S4, the control unit 11 displays the captured image Wp (wipe screen) on the display screen 13A. Specifically, the control unit 11 displays a partial image (captured image Wp) of a specific range including the detected user's face and hand, among the captured images of a predetermined area captured by the camera 15, at the end (corner) of the display screen 13A (see FIG. 2). The control unit 11 also displays a frame image representing a rectangular area A1 surrounding the user's hand and a frame image (operation area image) of the operation area A11 corresponding to the inputtable area F1 (see FIG. 2) of the display screen 13A, superimposed on the captured image Wp (see FIG. 3). Note that the display processing unit 111 may display the frame image of the operation area A11 shifted upward from its actual position with respect to the captured image Wp (see FIG. 11).

In step S5, the control unit 11 executes a process for setting a reference point P1 (reference point setting process). The control unit 11 executes a reference point setting process corresponding to any one of the first setting method, the second setting method, and the third setting method.

Figure 16 shows a specific example of the reference point setting process corresponding to the first setting method. Specifically, in step S11, the control unit 11 obtains the detection result of the hand detected from the captured image acquired by the camera 15. Next, in step S12, the control unit 11 determines the shape of the hand. For example, if the control unit 11 determines the shape (first shape) of the hand in an open state (paper state), the process proceeds to step S13. On the other hand, if the control unit 11 determines the shape (second shape) of the hand in a closed state (rock state), the process proceeds to step S121.

In step S13, the control unit 11 sets the center position of the bottom side L1 of the rectangular area A1 (see FIG. 4) as the reference point P1. After step S13, the process proceeds to step S6.

In step S121, the control unit 11 sets the previously calculated position of the first shape (par state) (see FIG. 6) as the reference point P1. After step S121, the process proceeds to step S6.

Figure 17 shows a specific example of the reference point setting process corresponding to the second setting method. Specifically, in step S21, the control unit 11 obtains the detection result of the hand detected from the captured image acquired by the camera 15. Next, in step S22, the control unit 11 determines the shape of the hand. For example, if the control unit 11 determines the shape (first shape) of the hand in an open state (paper state), the process proceeds to step S23. On the other hand, if the control unit 11 determines the shape (second shape) of the hand in a closed state (rock state), the process proceeds to step S221.

In step S23, the control unit 11 sets the position at a ratio of 1:2 from the end of the bottom side L1 of the rectangular area A1 on the side of the little finger of the user's hand (see FIG. 7) as the reference point P1. After step S23, the process proceeds to step S6.

In step S221, the control unit 11 sets the previously calculated position of the first shape (par state) as the reference point P1. After step S221, the process proceeds to step S6.

Figure 18 shows a specific example of a reference point setting process corresponding to the third setting method. Specifically, in step S31, the control unit 11 obtains a detection result of the hand detected from the captured image acquired by the camera 15. Next, in step S32, the control unit 11 determines the shape of the hand. For example, if the control unit 11 determines the shape (first shape) of an open hand (paper state), the process proceeds to step S33. On the other hand, if the control unit 11 determines the shape (second shape) of a closed hand (rock state), the process proceeds to step S321.

In step S33, the control unit 11 identifies specific pixels that display the color of the user's hand from among the multiple pixels contained in the rectangular area A1, and calculates the total number of the specific pixels for each pixel row in the column direction from the top side to the bottom side L1 of the rectangular area A1 (see FIG. 8). Next, in step S34, the control unit 11 extracts the pixel row on the bottom side L1 in which the total number of pixels exceeds a threshold value Cth. Next, in step S35, the control unit 11 calculates the coordinates of the minimum position Xmin and maximum position Xmax of the partial side corresponding to the extracted pixel row. Next, in step S36, the control unit 11 sets the center position Xc of the minimum position Xmin and maximum position Xmax to the reference point P1. After step S36, the process proceeds to step S6.

In step S321, the control unit 11 sets the previously calculated position of the first shape (par state) as the reference point P1. After step S321, the process proceeds to step S6.

In step S6 (see FIG. 15), the control unit 11 executes a process of calculating an input position corresponding to the reference point P1 (input position calculation process). The control unit 11 executes an input position calculation process corresponding to any one of the first calculation method, the second calculation method, and the third calculation method.

Figure 19 shows a specific example of the input position calculation process corresponding to the first calculation method. In the first calculation method, the control unit 11 displays a frame image of the operation area A11 on the captured image Wp (see Figure 12) in step S4 (see Figure 15).

In step S41, the control unit 11 sets a virtual inputtable area F2 that is larger than the inputtable area F1 (see FIG. 12). Next, in step S42, the control unit 11 maps the reference point P1 of the operation area A11 to the virtual inputtable area F2.

Next, in step S43, the control unit 11 determines whether the position mapped to the virtual inputtable area F2 is within the inputtable area F1. If the position mapped to the virtual inputtable area F2 is within the inputtable area F1 (S43: Yes), the process proceeds to step S45. On the other hand, if the position mapped to the virtual inputtable area F2 is not within the inputtable area F1 (S43: No), the process proceeds to step S44.

In step S44, the control unit 11 remaps the input area F1. Then, the process proceeds to step S45.

In step S45, the control unit 11 determines the mapped position as the input position. If the position mapped to the virtual inputtable area F2 is within the inputtable area F1 (S43: Yes), the control unit 11 determines that position as the input position. If the position mapped to the virtual inputtable area F2 is not within the inputtable area F1 (S43: No), the control unit 11 determines the edge position of the inputtable area F1 as the input position. The process then proceeds to step S7.

Figure 20 shows a specific example of the input position calculation process corresponding to the second calculation method. In the second calculation method, the control unit 11 displays a frame image of a virtual operation area B1 that is larger than the operation area A11 in the captured image Wp in step S4 (see Figure 15) (see Figure 13).

In step S51, the control unit 11 maps the reference point P1 of the operation area A11 to the inputtable area F1.

Next, in step S52, the control unit 11 determines whether the position mapped to the inputtable area F1 is within the inputtable area F1. If the position mapped to the inputtable area F1 is within the inputtable area F1 (S52: Yes), the process proceeds to step S54. On the other hand, if the position mapped to the inputtable area F1 is not within the inputtable area F1 (S52: No), the process proceeds to step S53.

In step S53, the control unit 11 remaps the input area F1. Then, the process proceeds to step S54.

In step S54, the control unit 11 determines the mapped position as the input position. If the position mapped to the input area F1 is within the input area F1 (S52: Yes), the control unit 11 determines that position as the input position. If the position mapped to the input area F1 is not within the input area F1 (S52: No), the control unit 11 determines the position at the edge of the input area F1 as the input position. The process then proceeds to step S7.

Figure 21 shows a specific example of an input position calculation process corresponding to the third calculation method. In the third calculation method, the control unit 11 displays a frame image of the virtual operation area B1 on the captured image Wp (see Figure 14) in step S4 (see Figure 15).

In step S61, the control unit 11 sets a virtual inputtable area F2 that is larger than the inputtable area F1 (see FIG. 14). Next, in step S62, the control unit 11 maps the reference point P1 of the virtual operation area B1 to the virtual inputtable area F2.

Next, in step S63, the control unit 11 determines whether the position mapped to the virtual inputtable area F2 is within the inputtable area F1. If the position mapped to the virtual inputtable area F2 is within the inputtable area F1 (S63: Yes), the process proceeds to step S65. On the other hand, if the position mapped to the virtual inputtable area F2 is not within the inputtable area F1 (S63: No), the process proceeds to step S64.

In step S64, the control unit 11 remaps the input area F1. Then, the process proceeds to step S65.

In step S65, the control unit 11 determines the mapped position as the input position. If the position mapped to the virtual inputtable area F2 is within the inputtable area F1 (S63: Yes), the control unit 11 determines that position as the input position. If the position mapped to the virtual inputtable area F2 is not within the inputtable area F1 (S63: No), the control unit 11 determines the edge position of the inputtable area F1 as the input position. The process then proceeds to step S7.

In step S7 (see FIG. 15), the control unit 11 executes input processing based on the determined input position. Specifically, the control unit 11 displays the mouse cursor M1 at the determined input position. Also, for example, if the determined input position is on the "Yes" button and the user closes his/her hand while the mouse cursor M1 is positioned on the "Yes" button, the control unit 11 accepts the "Yes" button operation and executes a predetermined process.

In step S8, the control unit 11 determines whether or not the input operation by the user has ended. If the input operation has ended (S8: Yes), the control unit 11 ends the display control process. If the input operation has not ended (S8: No), the control unit 11 returns to step S1. In this manner, the control unit 11 executes the display control process.

As described above, the display device 1 according to this embodiment detects a gesture operation made by the user's hand and performs input processing at an input position on the display screen. Specifically, the display device 1 acquires an image of the user, detects the user's hand from the acquired image, and determines the shape of the user's hand. The display device 1 also sets a reference point P1 at a predetermined position on the bottom side L1 of a specific area (rectangular area A1) that surrounds the detected user's hand, and if the shape of the detected user's hand is the predetermined shape, performs input processing with the set reference point P1 as the input position.

With this configuration, regardless of changes in the hand shape (gesture operation) (e.g., from a paper to a rock), the reference point P1 corresponding to the input position can be fixed. This makes it possible to prevent the input position from changing in a direction unintended by the user due to changes in the user's gesture operation.

The display device 1 according to this embodiment also acquires an image of the user, detects the user's hand from the acquired image, and sets an operation area A11 for gesture operation that is an area including the detected user's hand and corresponds to an inputtable area F1 in the display screen 13A where input can be made by a gesture operation. The display device 1 also displays the acquired image Wp on the display screen 13A, and displays an operation area image (frame image) corresponding to the set operation area A11 superimposed on the image Wp. For example, the display device 1 displays a frame image of the operation area A11 (see FIG. 12) or a virtual operation area B1 (see FIG. 13 and FIG. 14) on the image Wp.

With this configuration, by checking the frame image displayed on the captured image Wp, the user can easily grasp the correspondence between the range in which the user can move his or her hand and the movable range within the display screen 13A of the mouse cursor M1, which moves on the display screen 13A in response to the movement of the user's hand. This makes it possible to improve the operability of the display device 1.

As shown in the above-described embodiment, the present invention is a display device that includes a process for executing any one of the first to third setting methods and a process for executing any one of the first to third calculation methods. However, the present invention is not limited to the above-described embodiment, and may be a display device that includes at least a process for executing any one of the first to third setting methods. The present invention may also be a display device that includes at least a process for executing any one of the first to third calculation methods.

The display device of the present invention can be constructed by freely combining the above-described embodiments, or by appropriately modifying or partially omitting each embodiment, within the scope of the invention described in each claim.

1: Display device 11: Control unit 12: Storage unit 13: Display panel 13A: Display screen 14: Operation unit 15: Camera 111: Display processing unit 112: Acquisition processing unit 113: Detection processing unit 114: Determination processing unit 115: Reference point setting processing unit 116: Operation area setting processing unit 117: Virtual input possible area setting processing unit 118: Input processing unit

Claims (7)

An input device that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen,
an acquisition processing unit that acquires an image of the user;
a detection processing unit that detects the user's hand from the captured image acquired by the acquisition processing unit;
a first setting processing unit that sets an operation area of the gesture operation, the operation area including the user's hand detected by the detection processing unit, the operation area corresponding to an inputtable area on the display screen where input can be made by the gesture operation;
a second setting processing unit that sets a virtual inputtable area larger than the inputtable area;
a display processing unit that displays the captured image acquired by the acquisition processing unit on the display screen, and that displays an operation area image representing an outer frame of the operation area set by the first setting processing unit so as to be superimposed on the captured image , and further displays the operation area image shifted upward with respect to the captured image ;
a reference point setting processing unit that sets a reference point at a predetermined position on a lower side of the specific area including the user's hand detected by the detection processing unit;
an input processing unit that calculates, as the input position, a position in the virtual input possible area that corresponds to a position in the operation area that corresponds to the reference point set by the reference point setting processing unit, and executes the input process on the input position;
An input device comprising: An input device that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen,
an acquisition processing unit that acquires an image of the user;
a detection processing unit that detects the user's hand from the captured image acquired by the acquisition processing unit;
a first setting processing unit that sets an operation area of the gesture operation, the operation area including the user's hand detected by the detection processing unit, the operation area corresponding to an inputtable area on the display screen where input can be made by the gesture operation;
a display processing unit that displays the captured image acquired by the acquisition processing unit on the display screen, and displays an operation area image representing an outer frame of a virtual operation area larger than the operation area set by the first setting processing unit, superimposed on the captured image , and further shifts the operation area image upward with respect to the captured image;
a reference point setting processing unit that sets a reference point at a predetermined position on a lower side of the specific area including the user's hand detected by the detection processing unit;
an input processing unit that calculates, as the input position, a position in the input area that corresponds to the reference point set by the reference point setting processing unit and that is associated with the input area, and executes the input process on the input position;
An input device comprising: An input device that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen,
an acquisition processing unit that acquires an image of the user;
a detection processing unit that detects the user's hand from the captured image acquired by the acquisition processing unit;
a first setting processing unit that sets an operation area of the gesture operation, the operation area including the user's hand detected by the detection processing unit, the operation area corresponding to an inputtable area on the display screen where input can be made by the gesture operation;
a second setting processing unit that sets a virtual inputtable area larger than the inputtable area;
a display processing unit that displays the captured image acquired by the acquisition processing unit on the display screen, and displays an operation area image representing an outer frame of a virtual operation area larger than the operation area set by the first setting processing unit, superimposed on the captured image , and further shifts the operation area image upward with respect to the captured image;
a reference point setting processing unit that sets a reference point at a predetermined position on a lower side of the specific area including the user's hand detected by the detection processing unit;
an input processing unit that calculates, as the input position, a position in the virtual operation area corresponding to the reference point set by the reference point setting processing unit, which is associated with the virtual input possible area, and executes the input process on the input position;
An input device comprising: the display processing unit displays the operation area image on the display screen, shifted upward by half the width of the specific area in the vertical direction with respect to the captured image.
The input device according to any one of claims 1 to 3 . The specific area is a rectangular area surrounding the user's hand.
The input device according to any one of claims 1 to 3 . An input method for detecting a gesture operation by a user's hand and executing an input process at an input position on a display screen, comprising:
One or more processors
An acquisition step of acquiring a captured image of the user;
a detection step of detecting a hand of the user from the captured image acquired in the acquisition step;
a first setting step of setting an operation area of the gesture operation, the operation area including the hand of the user detected in the detection step and corresponding to an inputtable area on the display screen where an input can be made by the gesture operation;
a second setting step of setting a virtual inputtable area larger than the inputtable area;
a display step of displaying, on the display screen, the captured image acquired in the acquisition step, and displaying an operation area image representing an outer frame of the operation area set in the first setting step so as to be superimposed on the captured image , and further displaying the operation area image so as to be shifted upward with respect to the captured image;
a reference point setting step of setting a reference point at a predetermined position on a lower side of the specific area including the user's hand detected in the detection step;
an input step of calculating, as the input position, a position in the virtual input possible area that corresponds to a position in the operation area that corresponds to the reference point set in the reference point setting step, and executing the input process on the input position;
Input method to perform. An input program that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen,
An acquisition step of acquiring a captured image of the user;
a detection step of detecting a hand of the user from the captured image acquired in the acquisition step;
a first setting step of setting an operation area of the gesture operation, the operation area including the hand of the user detected in the detection step and corresponding to an inputtable area on the display screen where an input can be made by the gesture operation;
a second setting step of setting a virtual inputtable area larger than the inputtable area;
a display step of displaying, on the display screen, the captured image acquired in the acquisition step, and displaying an operation area image representing an outer frame of the operation area set in the first setting step so as to be superimposed on the captured image , and further displaying the operation area image so as to be shifted upward with respect to the captured image;
a reference point setting step of setting a reference point at a predetermined position on a lower side of the specific area including the user's hand detected in the detection step;
an input step of calculating, as the input position, a position in the virtual input possible area that corresponds to a position in the operation area that corresponds to the reference point set in the reference point setting step, and executing the input process on the input position;
An input program for execution by one or more processors.