JP7613716B2 - Registration device, operation device, registration method, operation method, program, and recording medium - Google Patents

Description

The present invention relates to a registration device, an operation device, a registration method, an operation method, a program, and a recording medium.

Technology has been reported that uses a motion capture system to recognize gestures expressed by a sequence of multiple user poses to operate an application (e.g., Patent Document 1, etc.)

JP 2015-116308 A

In the technology of Patent Document 1, a gesture selected by the user is registered from among candidate gestures generated based on a human body motion model database. However, there is a demand for technology that allows users to register their own poses and create gestures from those poses.

The present invention aims to provide a registration device, an operation device, a registration method, an operation method, a program, and a recording medium that allow a user to register their own poses and create gestures from those poses.

In order to achieve the above object, the registration device of the present invention comprises:
A pose registration unit and a gesture registration unit are included,
the pose registration unit includes a pose image acquisition unit, a pose setting unit, and a registration unit;
the pose image acquisition unit acquires a registration pose image including skeleton information;
The pose setting unit sets pose information based on the skeleton information,
The registration unit registers the set pose information,
the gesture registration unit includes an input receiving unit, a gesture setting unit, and a registration unit;
the input receiving unit receives an input of gesture creation information;
The gesture creation information is information regarding creation of a gesture, and includes information regarding pose information selected by a user from among one or more of the pose information;
the gesture setting unit sets gesture information including one or more pieces of pose information based on the gesture creation information;
The registration unit is a device that registers the set gesture information.

The operating device of the present invention comprises:
A registration unit, a gesture image acquisition unit, a recognition unit, and an operation control unit are included.
the registration unit uses a registration device of the present invention including an operation information registration unit to register pose information set based on the skeletal information, the gesture information configured from one or more pieces of pose information, and the operation information related to an operation of an application linked to the gesture information;
the gesture image acquisition unit acquires a gesture image composed of one or more pose images including the skeleton information,
The recognition unit recognizes the registered gesture information from the gesture image,
The operation control unit is a device that controls an input device to operate an application based on the operation information linked to the recognized gesture information.

The registration method of the present invention comprises the steps of:
The method includes a pose registration step and a gesture registration step,
The pose registration step includes a pose image acquisition step, a pose setting step, and a registration step,
The pose image acquisition step acquires a registration pose image including skeleton information,
The pose setting step sets pose information based on the skeleton information,
The registration step includes registering the set pose information,
The gesture registration step includes an input receiving step, a gesture setting step, and a registration step,
The input receiving step receives an input of gesture creation information;
The gesture creation information is information regarding creation of a gesture, and includes information regarding pose information selected by a user from among one or more of the pose information;
The gesture setting step sets gesture information including one or more pieces of pose information based on the gesture creation information,
The registration step is a method for registering the set gesture information.

The method of operation of the present invention comprises:
The method includes a registration process, a gesture image acquisition process, a recognition process, and an operation control process,
The registration step includes a registration method according to an aspect of the present invention including an operation information registration step, which registers pose information set based on the skeletal information, the gesture information composed of one or more pieces of pose information, and the operation information related to an operation of an application linked to the gesture information;
The gesture image acquiring step acquires a gesture image including one or more pose images including the skeleton information,
The recognizing step recognizes the registered gesture information from the gesture image,
The operation control step is a method of controlling an input device to operate an application based on the operation information linked to the recognized gesture information.

The present invention allows users to register poses as they wish, and enables them to create gestures from those poses.

FIG. 1 is a block diagram illustrating an example of a configuration of a registration device according to the first embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the registration device according to the first embodiment. FIG. 3 is a flowchart illustrating an example of processing in the registration device according to the first embodiment. FIG. 4 is a block diagram illustrating an example of a configuration of an operation device according to the second embodiment. FIG. 5 is a block diagram illustrating an example of a hardware configuration of the operation device according to the second embodiment. FIG. 6 is a flowchart illustrating an example of a process in the operation device according to the second embodiment. FIG. 7 is a diagram showing an example of capturing an image of a pose displayed on a display in the third embodiment. FIG. 8 is a diagram showing an example of setting pose information based on skeleton information and used movable connecting part information in the third embodiment. FIG. 9 is a diagram showing an example of gesture information to be registered in the third embodiment. FIG. 10 is a diagram showing an example of gesture information to be registered in the third embodiment. FIG. 11 is a diagram showing an example of gesture information to be registered in the third embodiment. FIG. 12 is a diagram showing an example in which the operating device of the fourth embodiment is applied to the operation of a large machine tool in a factory. FIG. 13 is a diagram showing an example in which the operation device according to the fourth embodiment is applied to the operation of a monitor image in an operating room. FIG. 14 is a diagram showing an example in which the operation device according to the fourth embodiment is applied to the operation of a digital signage or a guide board.

In the registration device of the present invention, for example,
The pose registration unit further includes a pose request unit and a recognition unit,
the pose request unit requests capturing of a test pose image including a target for authentication assuming a pose indicated by the pose information;
the pose image acquisition unit acquires a test pose image including the skeleton information,
The recognition unit recognizes the set pose information from the test pose image,
The registration unit may be configured to register the pose information when the pose information is recognized.

In the registration device of the present invention, for example,
the gesture registration unit further includes a gesture request unit, a gesture image acquisition unit, and a recognition unit;
the gesture request unit requests capturing of a test gesture image including the authentication target making a gesture indicated by the gesture information;
the gesture image acquisition unit acquires a test gesture image composed of one or more of the pose images including the skeleton information;
The recognition unit recognizes the set gesture information from the test gesture image,
The registration unit may register the gesture information when the gesture information is recognized.
In the above aspect, for example,
the gesture request unit causes a display device to display the one or more pieces of pose information constituting the gesture information;
The recognition unit may display, for each piece of pose information, visual information indicating whether or not the pose information has been recognized, superimposed on the pose information displayed on the display device.

In the registration device of the present invention, for example,
The pose registration unit further includes an input reception unit,
the input receiving unit receives input of information on a movable link to be used for a pose based on the skeletal information;
The pose setting unit may set pose information based on the skeleton information and the used movable link information.
In the above aspect, for example, the pose setting unit may display the skeletal information on a display device, and superimpose visual information indicating the used movable connecting part information for which the input has been received on the skeletal information displayed on the display device.

In the registration device of the present invention, for example,
The gesture creation information may further include control information relating to control of the pose information.

The registration device of the present invention includes, for example,
Further, an operation information registration unit is included,
The operation information registration unit may register operation information relating to an operation of an application in association with the registered gesture information.

In the registration method of the present invention, for example,
The pose registration step further includes a pose request step and a recognition step,
The pose request step requests capturing of a test pose image including a target for authentication assuming a pose indicated by the pose information,
The pose image acquisition step acquires a test pose image including the skeleton information,
the recognizing step recognizes the set pose information from the test pose image,
The registration step may be such that, if the pose information is recognized, the pose information is registered.

In the registration method of the present invention, for example,
The gesture registration step further includes a gesture request step, a gesture image acquisition step, and a recognition step.
The gesture request step requests capturing of a test gesture image including the authentication target making a gesture indicated by the gesture information,
The gesture image acquiring step acquires a test gesture image composed of one or more of the pose images including the skeleton information,
The recognizing step recognizes the set gesture information from the test gesture image,
The registration step may be performed in such a manner that, when the gesture information is recognized, the gesture information is registered.
In the above aspect, for example, the gesture request step includes displaying the one or more pieces of pose information constituting the gesture information on a display device;
The recognition step may be performed in a mode in which visual information indicating whether or not the pose information has been recognized is displayed, for each piece of pose information, superimposed on the pose information displayed on the display device.

In the registration method of the present invention, for example,
The pose registration step further includes an input receiving step,
The input receiving step receives input of information on a movable connecting step to be used for a pose based on the skeleton information;
The pose setting step may include setting pose information based on the skeleton information and the movable connection step information.
In the above aspect, the pose setting process may include displaying the skeletal information on a display device, and superimposing visual information indicating the used movable connecting part information for which the input has been received on the skeletal information displayed on the display device.

In the registration method of the present invention, for example,
The gesture creation information may further include control information relating to control of the pose information.

The registration method of the present invention includes, for example,
Further, the method includes an operation information registration step,
The operation information registration step may include registering operation information relating to an operation of an application in association with the registered gesture information.

The program of the present invention is a program for causing a computer to execute each step of the method of the present invention as a procedure.

The recording medium of the present invention is a computer-readable recording medium on which the program of the present invention is recorded.

In the present invention, a "pose" refers to a posture taken by the authentication subject. The pose may be, for example, a pose taken consciously or unconsciously by the authentication subject. In the present invention, a "gesture" refers to a series of movements consisting of one or more of the poses. In the present invention, the "authentication subject" is not particularly limited and may be, for example, a human, a non-human animal, a robot, etc.

Next, an embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiment. In each of the drawings, the same parts are given the same reference numerals. Furthermore, the explanations of each embodiment can be mutually incorporated unless otherwise specified, and the configurations of each embodiment can be combined unless otherwise specified.

[Embodiment 1]
FIG. 1 is a block diagram showing an example of a configuration of a registration device 100 according to the present embodiment. As shown in FIG. 1, the device 100 includes a pose registration unit 10 and a gesture registration unit 20. The device 100 may further include, for example, an operation information registration unit 30 and an imaging unit 40 as an optional configuration. The pose registration unit 10 includes a pose image acquisition unit 11, a pose setting unit 12, and a registration unit 15. The pose registration unit 10 may further include, for example, a pose request unit 13, a recognition unit 14, and an input acceptance unit 16 as an optional configuration. The gesture registration unit 20 includes an input acceptance unit 21, a gesture setting unit 22, and a registration unit 26. The gesture registration unit 20 may further include, for example, a gesture request unit 23, a gesture image acquisition unit 24, and a recognition unit 25 as an optional configuration.

The device 100 may be, for example, a single device including each of the above-mentioned parts, or a device to which each of the above-mentioned parts can be connected via a communication line network. The device 100 may also be connected to an external device described later via the communication line network. The communication line network is not particularly limited and may be a known network, for example, wired or wireless. Examples of the communication line network include the Internet line, WWW (World Wide Web), telephone line, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (Local 5G), etc. Examples of wireless communication include WiFi (Wireless Fidelity), Bluetooth (registered trademark), local 5G, and LPWA. The wireless communication may be a form in which each device communicates directly (Ad Hoc communication), infrastructure communication, indirect communication via an access point, and the like. The device 100 may be incorporated into a server as a system, for example. The device 100 may be, for example, a personal computer (PC, for example, desktop type or notebook type) in which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. Furthermore, the device 100 may be, for example, a form of cloud computing or edge computing in which at least one of the parts is on a server and the other parts are on a terminal.

Figure 2 shows an example block diagram of the hardware configuration of the device 100. The device 100 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device 106, a communication device 107, etc. Furthermore, the hardware configuration of the device 100 may further include, for example, an imaging device 108 as an optional configuration. Each part of the device 100 is connected to each other via the bus 103 by their respective interfaces (I/F).

The central processing unit 101 is responsible for the overall control of the device 100. In the device 100, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various information. Specifically, for example, the central processing unit 101 functions as a pose registration unit 10 and a gesture registration unit 20. The central processing unit 101 also functions, for example, as an operation information registration unit 30.

The bus 103 can also be connected to, for example, an external device. Examples of the external device include an external storage device (external database, etc.), a printer, an external input device, an external display device, an external imaging device, etc. The present device 100 can be connected to an external network (the above-mentioned communication line network) by, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

The memory 102 may be, for example, a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operating programs, such as the program of the present invention, stored in the storage device 104 described below, and the central processing unit 101 receives data from the memory 102 and executes the program. The main memory may be, for example, a RAM (random access memory). The memory 102 may also be, for example, a ROM (read only memory).

The storage device 104 is also referred to as an auxiliary storage device, for example, in contrast to the main memory. As described above, the storage device 104 stores operating programs including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive that reads and writes from the recording medium. The recording medium is not particularly limited, and may be, for example, an internal or external type, such as a hard disk drive (HDD), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, etc. The storage device 104 may be, for example, a hard disk drive (HDD) in which the recording medium and the drive are integrated, or a solid state drive (SSD).

In the device 100, the memory 102 and storage device 104 can also store user log information as well as information obtained from an external database (not shown).

The device 100 may further include, for example, an input device 105, a display device 106, and an imaging device 108. The input device 105 is, for example, a touch panel, a keyboard, a mouse, etc. The display device 106 is, for example, an LED display, a liquid crystal display, etc. The imaging device 108 is, for example, a camera (2D camera, 3D camera, etc.), etc., and functions as the imaging unit 40.

Next, an example of the registration method of this embodiment will be described based on the flowchart in FIG. 3. The registration method of this embodiment is implemented, for example, as follows using the registration device 100 in FIG. 1. Note that the registration method of this embodiment is not limited to the use of the registration device 100 in FIG. 1.

First, a pose is registered by the pose registration unit 10 (S10). Then, a gesture is registered by the gesture registration unit 20 (S20), and the process ends (END).

The processing of the pose registration unit 10 will be specifically described. First, the pose image acquisition unit 11 acquires a registration pose image including skeletal information (S11). The skeletal information includes, for example, position information (for example, three-dimensional position information) of each movable joint of the authentication target and skeletal line information. The skeletal line information is, for example, information obtained by connecting each movable joint with a line along the configuration of the authentication target. Specifically, when the authentication target is a human, the information includes position information of each joint of the body and skeletal line information connecting each joint along the configuration of the body. The registration pose image may be, for example, a still image or a video. The pose image acquisition unit 11 may acquire the registration pose image from an external device, for example, via the communication line network. When the present device 100 includes an imaging unit 40, the pose image acquisition unit 11 may acquire, for example, the registration pose image captured by the imaging unit 40. Additionally, the device 100 may acquire the registration pose image using, for example, a known motion capture technology (such as Nuitrack by 3DiVi, Inc.).

Next, the pose setting unit 12 sets pose information based on the skeletal information (S12). The pose information is, for example, information indicating the relationship between the skeletal information and the pose. In addition, when the pose registration unit 10 includes an input receiving unit 16, for example, as shown in FIG. 3, after the step (S11), the input receiving unit 16 may receive input of movable joint information to be used for the pose based on the skeletal information (S17). Then, in the step (S12), the pose setting unit 12 may set pose information based on the skeletal information and the movable joint information to be used. The "movable joint to be used" means, for example, the position of the movable joint to be used for the pose, and the position of the movable joint that is the point of the pose. Specifically, the movable joint to be used preferably includes, for example, a movable joint with little movement (for example, the head, neck, or torso in the case of a joint) and a movable joint that is the end of the movement (for example, the wrist or ankle in the case of a joint). The input reception unit 16 may receive the used movable link information input by the user via an input device (such as the input device 105 or an external input device), or may receive the input of the used movable link information acquired by the device 100 from the outside via the communication line network. The pose setting unit 12 may, for example, display the skeletal information on a display device, and superimpose visual information indicating the used movable link information for which the input has been received on the skeletal information displayed on the display device. The display device is not particularly limited, and may be, for example, an external display device or the display device 106. The visual information is, for example, a color, a symbol, a figure, or the like. Specifically, for example, in the movable link of the skeletal information displayed on the display device, the color or figure of the movable link indicated by the used movable link information is displayed in a color (for example, green) or shape (for example, triangle) different from the color (for example, white) or shape (for example, circle) of the movable link not indicated by the used movable link information. This makes it easier for users to visually understand which movable joints are used for poses.

Next, for example, the pose request unit 13 may request capturing a test pose image including the authentication target assuming the pose indicated by the pose information (S13). The request may be made to, for example, an external imaging device, or to the imaging device 108 included in the device 100.

Next, for example, the pose image acquisition unit 11 may acquire a test pose image including the skeletal information (S14). The test pose image may be, for example, a still image or a video. The pose image acquisition unit 11 may acquire the test pose image from an external device, for example, via the communication line network. If the device 100 includes an imaging unit 40, the pose image acquisition unit 11 may acquire, for example, the test pose image captured by the imaging unit 40. Furthermore, the device 100 may acquire the test pose image using, for example, a known motion capture technology (for example, Nuitrack by 3DiVi, etc.).

Next, for example, the recognition unit 14 recognizes the set pose information from the test pose image (S15). When the pose setting unit 12 sets the pose information based on the skeletal information and the used movable link information, the recognition unit 14 may recognize the set pose information based on the movable link indicated by the used movable link information, for example, from the skeletal information of the test pose image.

Note that steps (S13) to (S15) are optional steps, and step (S16) described below may be performed immediately after step (S12) without performing these steps.

Then, the registration unit 15 registers the pose information (S16), and the processing of the pose registration unit 10 is terminated. When recognition is performed by the recognition unit 14, for example, the pose information may be registered if the pose information is recognized. The pose information may be registered, for example, in the memory 102 and the storage device 104, or in an external storage device.

The processing of the gesture registration unit 20 will be specifically described. First, the input acceptance unit 21 accepts input of gesture creation information (S21). The gesture creation information is information related to gesture creation, and includes information related to pose information selected by the user from among the one or more pose information. The "one or more pose information" may include not only the pose information registered by the pose registration unit 10 in the step (S10), but also, for example, pose information registered (stored) in advance in the present device 100. The information related to the pose information selected by the user may include, for example, information related to the order of the pose information. The gesture information may further include, for example, control information related to control of the pose information. The control information may include, for example, information related to repetition of a pose, a time limit related to at least one of the gesture and the pose, and the like. The input acceptance unit 21 may accept, for example, the gesture creation information input by the user via an input device (such as the input device 105 or an external input device), or may accept the input of the gesture creation information acquired by the present device 100 from the outside via the communication line network.

Next, the gesture setting unit 22 sets gesture information consisting of one or more pose information based on the gesture creation information (S22).

Next, for example, the gesture request unit 23 may request capturing an image of a test gesture image including the authentication target making the gesture indicated by the gesture information (S23). The request may be made to, for example, an external imaging device, or to the imaging device 108 included in the device 100.

Next, for example, the gesture image acquisition unit 24 may acquire a test gesture image composed of one or more pose images including the skeletal information (S24). The test gesture image may be, for example, a still image or a video. The gesture image acquisition unit 24 may acquire the test gesture image from an external device via the communication line network. If the device 100 includes an imaging unit 40, the gesture image acquisition unit 24 may acquire the test gesture image captured by the imaging unit 40. The device 100 may also acquire the test gesture image using, for example, a known motion capture technology (for example, Nuitrack by 3DiVi, etc.).

Next, for example, the recognition unit 25 may recognize the set gesture information from the test gesture image (S25). Specifically, the recognition unit 25 recognizes the gesture information by, for example, recognizing one or more pieces of pose information. In other words, when there are two or more pieces of pose information, the recognition unit 25, for example, repeatedly executes the process by the recognition unit 14. The specific process for recognizing the pose information is the same as that of the recognition unit 14.

Note that steps (S23) to (S25) are optional steps, and step (S26) described below may be performed immediately after step (S22) without performing these steps.

Then, the registration unit 26 registers the gesture information (S26), and the processing of the gesture registration unit 20 is terminated. When recognition is performed by the recognition unit 25, for example, if the gesture information is recognized, the pose information may be registered. The gesture information may be registered, for example, in the memory 102 and the storage device 104, or in an external storage device.

In the step (S15) and the step (S25), if the recognition units 14 and 25 are unable to recognize the pose information and the gesture information, for example, the steps (S13) to (S15) and the steps (S23) to (S25) may be repeatedly executed. In addition, the recognition units 14 and 25 may notify a display device (external display device and display device 106, etc.) or an external device (for example, a user's terminal (PC, smartphone, tablet terminal, etc.)) of recognition information indicating at least one of the cases where the pose information and the gesture information can be recognized and where the recognition units 14 and 25 are unable to recognize the pose information and the gesture information, for example. Furthermore, the recognition units 14 and 25 may recognize the pose information and the gesture information within a preset time limit. In this case, the recognition units 14 and 25 process the pose information and the gesture information as being unable to be recognized, for example, if the time limit is exceeded. The time limit can be set arbitrarily, and in the gesture information, for example, it may be a time limit for the entire gesture information, or a time limit for each piece of pose information constituting the gesture information.

In the step (S23), the gesture request unit 23 may, for example, further display the one or more pieces of pose information constituting the gesture information on a display device. The display device is not particularly limited and may be an external display device or the display device 106. In the step (S25), the recognition unit 25 may, for example, superimpose visual information indicating whether or not the pose information has been recognized for each piece of pose information on the display device and display the visual information on the display device. The visual information may be visually identifiable information that differs for each result of the recognition, for example, characters, symbols, figures, etc. As a more specific example, for example, there is a frame surrounding each piece of pose information displayed on the display device, and the color of the frame differs depending on whether the piece of pose information has been recognized or not. This allows the user (for example, the authentication target) to know, for example, whether or not each piece of pose information has been recognized, and which piece of pose information has not been recognized when gesture information has not been recognized.

As described above, the present device 100 may further include, for example, an operation information registration unit 30. As shown in FIG. 3, following the step (S20), the operation information registration unit 30 registers operation information related to the operation of the application in association with the registered gesture information (S30). The operation information (also called an event) may be, for example, a mouse event or a key event. Examples of the mouse event include a click operation, a pointer movement operation, a wheel operation (scroll operation), etc. Examples of the key event include input operations of various keys on a keyboard. This allows, for example, the present device 100 to be applied to an operation device. The operation device is not particularly limited, but is a device capable of performing various operations according to the action of the authentication target, and specifically, for example, is the operation device 200 described in embodiment 2.

As described above, the device 100 may further include, for example, an imaging unit 40. The imaging unit 40 captures each image, for example, before the step (S11), the step (S14), and the step (S24) (not shown in FIG. 3).

According to this embodiment, a user's own pose can be registered by processing in the pose registration unit 10. Also, according to this embodiment, it is possible to create a gesture from one or more poses including the pose registered by processing in the pose registration unit 10. Furthermore, according to this embodiment, before registering the pose information and the gesture information, a test pose image and a test gesture image are obtained to test whether the pose information and the gesture information can be recognized, thereby preventing erroneous registration of the pose information and the gesture information. Also, in this embodiment, for example, by setting the pose information based on the used movable connecting part, it is possible to prevent registration of pose information unintended by the user and improve the accuracy of recognition by the recognition units 14 and 25. Furthermore, in this embodiment, for example, the used movable connecting part information can be superimposed and displayed on the test pose image and the test gesture image.

[Embodiment 2]
Fig. 4 is a block diagram showing an example of a configuration of the operation device 200 according to the present embodiment. As shown in Fig. 4, the device 200 includes a registration unit 201, a gesture image acquisition unit 202, a recognition unit 203, and an operation control unit 204. The device 200 may further include, for example, an imaging unit 205 as an optional configuration.

The device 200 may be, for example, a single device including the above-mentioned units, or a device to which the above-mentioned units can be connected via the communication line network. The device 200 may also be connectable to the above-mentioned external device via the communication line network. The communication line network is, for example, the same as described above. The device 200 may be, for example, incorporated into a server as a system. The device 200 may also be, for example, a personal computer (PC, for example, desktop type or notebook type) in which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. Furthermore, the device 200 may be in the form of cloud computing, edge computing, or the like, in which at least one of the above-mentioned units is on a server and the other units are on a terminal.

Figure 5 shows an example block diagram of the hardware configuration of the device 200. Unless otherwise specified, the hardware configuration of the device 200 is similar to the hardware configuration of the registration device 100 shown in Figure 2.

The central processing unit 101 functions, for example, as a registration unit 201, a gesture image acquisition unit 202, a recognition unit 203, and an operation control unit 204.

The imaging device 108 functions, for example, as an imaging unit 205.

Next, an example of the operation method of this embodiment will be described based on the flowchart of FIG. 6. The operation method of this embodiment is performed, for example, as follows using the operation device 200 of FIG. 4. Note that the operation method of this embodiment is not limited to the use of the operation device 200 of FIG. 4.

First, the registration unit 201 uses the registration device 100 including the operation information registration unit 30 described in the first embodiment to register the pose information set based on the skeletal information, the gesture information composed of one or more pieces of pose information, and the operation information related to the operation of an application linked to the gesture information (S201).

Next, the gesture image acquisition unit 202 acquires a gesture image composed of one or more pose images including the skeletal information (S202). Specifically, for example, this is the same as the processing in the gesture image acquisition unit 24 of the registration device 100. The gesture image acquisition unit 202 may use, for example, the gesture image acquisition unit 24 of the registration device 100.

Next, the recognition unit 203 recognizes the registered gesture information from the gesture image (S203). Specifically, for example, this is the same as the processing in the recognition unit 25 of the registration device 100. The recognition unit 203 may use, for example, the recognition unit 25 of the registration device 100.

Then, the operation control unit 204 controls the input device based on the operation information linked to the recognized gesture information to operate the application (S204), and ends (END). The input device may be, for example, an external input device or the input device 105. The application is not particularly limited, and may be, for example, an application stored in the memory 102 and the storage device 104, and may be an operating system or an application other than the operating system.

As described above, the device 200 may further include, for example, an imaging unit 205. The imaging unit 205 captures each image, for example, during the processing in the step (S201) and before the step (S202) (not shown in FIG. 6).

According to this embodiment, as in the first embodiment, processing in the registration unit 201 allows a user to register unique pose information, making it possible to create gestures from one or more poses. Furthermore, according to this embodiment, the created gesture can be used to operate an application without using an input device. Not using an input device, i.e., enabling non-contact operation, can provide effects such as improved work efficiency, reduced hygiene risks, and prevention of damage to the input device.

[Embodiment 3]
An example of registering poses and gestures made by a person to be authenticated will be described with reference to FIGS.

Figure 7 is a diagram showing an example of capturing a pose image displayed on a display. The pose image may be a pose image for registration or a pose image for testing. In Figure 7, display 1 is a touch display. As shown in Figure 7, a preview image of the pose image is displayed on display 1. The preview image includes skeletal information, similar to the pose image, as shown in Figure 7, for example. The skeletal information may be, for example, the entire body, or only the lower body, only the upper body, or only half the body. In the example shown in Figure 7, an image is captured of the user's front view (the same applies to other figures).

The pose image is captured, for example, according to the following steps (1) to (5).
(1) Press the “Start Imaging” button displayed on the display 1.
(2) A countdown to imaging begins, and the countdown is displayed on the display 1.
(3) The user strikes a pose while checking the pose in the preview image.
(4) When the count reaches 0, imaging begins, and the user remains still in the pose during imaging.
(5) Imaging ends.

Fig. 8 is a diagram showing an example of setting pose information based on the skeletal information and the used movable connecting part information. In the skeletal information shown in Fig. 8, joints (movable connecting parts) are indicated by circles. Specifically, for example, the pose information is set according to the following steps (1) to (5).
(1) Skeleton information contained in a captured pose image is displayed on the display 1.
(2) The user selects the joints to be used in the pose.
(3) In the skeletal information, selected joints and non-selected joints are displayed on the display 1 in a distinguishable manner (for example, by filling the circles with different colors).
(4) Setting pose information based on the skeleton information and the selected joints.
(5) Setting complete.

For example, as shown in FIG. 8, the registration device 100 may retake the pose image by pressing "take again" displayed on the display 1.

As shown in FIG. 8, if the pose has a point at the hand position, by using the movable link information in use, the registration device 100 can recognize it as the registered pose information, even if, for example, the foot position in the test pose image is different from the registered pose information, as long as the hand position is approximately the same as the registered pose information.

After setting the pose information, for example, as shown in FIG. 8, by pressing "Test" displayed on the display 1, a test pose image is requested by the pose request unit 13, a test pose image is acquired by the pose image acquisition unit 11, and recognition is performed by the recognition unit 14. Then, if the pose information is recognized, the registration unit 15 registers the pose information. The pose information may include, for example, a unique name for each pose. The registration device 100 may be capable of registering an arbitrary name for each pose, for example. A result indicating whether the pose information was recognized or not is displayed, for example, on the display 1.

9 to 11 are examples of gesture information to be registered. For example, the gesture "bye-bye" shown in FIG. 9 is composed of a pose in which the head is fixed, the left hand and left elbow are lowered, and the right hand and right elbow are raised diagonally to the right, and a pose in which the right hand and right elbow are raised above the head, repeated twice. The gesture "downward flick" shown in FIG. 9 is composed of a series of movements including a pose in which the head is fixed, the left hand and left elbow are lowered, and the right hand and right elbow are raised above the head, a pose in which the right hand and right elbow are stretched out to the right, and a pose in which the right hand and right elbow are lowered. The gesture "left flick" shown in FIG. 10 is composed of a series of movements including a pose in which the head is fixed, the right hand and right elbow and the left hand and left elbow are lowered, and a pose in which the right hand and right elbow are raised above the head and the left hand and left elbow are stretched out to the left. The gesture "right flick" shown in FIG. 10 is made up of a series of movements including a pose with the head fixed, the right hand and right elbow lowered, and the left hand and left elbow lowered, and a pose with the right hand and right elbow extended to the right side and the left hand and left elbow raised above the head. The gesture "upward flick" shown in FIG. 11 is made up of a series of movements including a pose with the head fixed, the left hand and left elbow lowered, the right hand and right elbow extended to the right side, and a pose with the right hand and right elbow raised above the head.

[Embodiment 4]
An example in which the present invention is applied to each scene will be described with reference to FIGS.

Figure 12 shows an example of the application of the operating device 200 to the operation of a large machine tool in a factory. Before the operating device 200 was introduced, as shown in the left diagram of Figure 12, the worker had to move to the input device every time he performed an input operation, which resulted in a problem of the worker's long lead line. In addition, the worker had to take the time to put on and take off gloves. However, after the operating device 200 was introduced, the worker no longer had to move to the input device, and could perform input operations by making gestures on the spot. This shortens the lead line for the worker, and also eliminates the time to put on and take off gloves. In Figure 12, the gesture is captured by a 3D camera, and the captured data is sent to the operating device 200 via the communication network (similar to Figures 13 and 14).

Figure 13 shows an example of the application of the operation device 200 to the operation of a monitor image in an operating room. Before the introduction of the operation device 200, as shown in the left diagram of Figure 13, an assistant operated the monitor screen to enable the viewing of MRI images, etc. However, after the introduction of the operation device 200, there is no need to assign an assistant, and the doctor can operate the monitor screen by making gestures on the spot. This makes it possible to reduce labor costs.

Figure 14 shows an example of the application of the operating device 200 to the operation of digital signage or information boards. Before the operating device 200 was introduced, as shown in the left diagram of Figure 14, many people would touch the digital signage to operate it (contact operation), which raised the risk of damage or destruction of the digital signage. In addition, the contact operation increases the risk of infection with bacteria, viruses, etc. However, after the operating device 200 was introduced, operation by gestures, i.e., non-contact operation, is possible. This eliminates the risk of damage or destruction, and reduces the risk of infection.

[Embodiment 5]
The program of the present embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, the "procedure" may be read as "processing." The program of the present embodiment may be recorded, for example, in a computer-readable recording medium. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), and an optical disk.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

<Additional Notes>
Some or all of the above embodiments may be described as follows, but are not limited to the following supplementary notes.
(Appendix 1)
A pose registration unit and a gesture registration unit are included,
the pose registration unit includes a pose image acquisition unit, a pose setting unit, and a registration unit;
the pose image acquisition unit acquires a registration pose image including skeleton information;
The pose setting unit sets pose information based on the skeleton information,
The registration unit registers the set pose information,
the gesture registration unit includes an input receiving unit, a gesture setting unit, and a registration unit;
the input receiving unit receives an input of gesture creation information;
the gesture creation information is information regarding creation of a gesture, and includes information regarding pose information selected by a user from among the plurality of pieces of pose information;
the gesture setting unit sets gesture information including one or more pieces of pose information based on the gesture creation information;
The registration unit registers the set gesture information.
(Appendix 2)
The pose registration unit further includes a pose request unit and a recognition unit,
the pose request unit requests capturing of a test pose image including a target for authentication assuming a pose indicated by the pose information;
the pose image acquisition unit acquires a test pose image including the skeleton information,
The recognition unit recognizes the set pose information from the test pose image,
2. The registration device according to claim 1, wherein the registration unit registers the pose information when the pose information is recognized.
(Appendix 3)
the gesture registration unit further includes a gesture request unit, a gesture image acquisition unit, and a recognition unit;
the gesture request unit requests capturing of a test gesture image including the authentication target making a gesture indicated by the gesture information;
the gesture image acquisition unit acquires a test gesture image composed of one or more of the pose images including the skeleton information;
The recognition unit recognizes the set gesture information from the test gesture image,
3. The registration device according to claim 1, wherein the registration unit registers the gesture information when the gesture information is recognized.
(Appendix 4)
the gesture request unit causes a display device to display the one or more pieces of pose information constituting the gesture information;
The registration device according to claim 3, wherein the recognition unit displays, for each piece of pose information, visual information indicating whether or not the pose information has been recognized, superimposed on the pose information displayed on the display device.
(Appendix 5)
The pose registration unit further includes an input reception unit,
the input receiving unit receives input of information on a movable link to be used for a pose based on the skeletal information;
The registration device according to any one of appendices 1 to 4, wherein the pose setting unit sets pose information based on the skeletal information and the used movable connecting part information.
(Appendix 6)
The registration device described in Appendix 5, wherein the pose setting unit displays the skeletal information on a display device, and superimposes visual information indicating the used movable connecting part information for which the input has been received on the skeletal information displayed on the display device.
(Appendix 7)
7. The registration device according to claim 1, wherein the gesture creation information further includes control information regarding control of the pose information.
(Appendix 8)
Further, an operation information registration unit is included,
8. The registration device according to claim 1, wherein the operation information registration unit registers operation information related to an operation of an application in association with the registered gesture information.
(Appendix 9)
A registration unit, a gesture image acquisition unit, a recognition unit, and an operation control unit are included.
the registration unit uses a registration device described in Supplementary Note 8 to register pose information set based on the skeletal information, the gesture information configured from one or more pieces of pose information, and the operation information related to an operation of an application linked to the gesture information;
the gesture image acquisition unit acquires a gesture image composed of one or more pose images including the skeleton information;
The recognition unit recognizes the registered gesture information from the gesture image,
The operation control unit controls an input device to operate an application based on the operation information linked to the recognized gesture information.
(Appendix 10)
The method includes a pose registration step and a gesture registration step,
The pose registration step includes a pose image acquisition step, a pose setting step, and a registration step,
The pose image acquisition step acquires a registration pose image including skeleton information,
The pose setting step sets pose information based on the skeleton information,
The registration step includes registering the set pose information,
The gesture registration step includes an input receiving step, a gesture setting step, and a registration step,
The input receiving step receives an input of gesture creation information;
The gesture creation information is information regarding creation of a gesture, and includes information regarding pose information selected by a user from among one or more of the pose information;
The gesture setting step sets gesture information including one or more pieces of pose information based on the gesture creation information,
The registration step registers the set gesture information.
(Appendix 11)
The pose registration step further includes a pose request step and a recognition step,
The pose request step requests capturing of a test pose image including a target for authentication assuming a pose indicated by the pose information,
The pose image acquisition step acquires a test pose image including the skeleton information,
the recognizing step recognizes the set pose information from the test pose image,
11. The registration method according to claim 10, wherein the registration step registers the pose information if the pose information is recognized.
(Appendix 12)
The gesture registration step further includes a gesture request step, a gesture image acquisition step, and a recognition step.
The gesture request step requests capturing of a test gesture image including the authentication target making a gesture indicated by the gesture information,
The gesture image acquiring step acquires a test gesture image composed of one or more of the pose images including the skeleton information,
The recognizing step recognizes the set gesture information from the test gesture image,
12. The method according to claim 10, wherein the registration step registers the gesture information if the gesture information is recognized.
(Appendix 13)
The gesture request step displays the one or more pieces of pose information constituting the gesture information on a display device,
The registration method according to claim 12, wherein the recognition step displays, for each piece of pose information, visual information indicating whether or not the pose information has been recognized, superimposed on the pose information displayed on the display device.
(Appendix 14)
The pose registration step further includes an input receiving step,
The input receiving step receives input of information on a movable connecting step to be used for a pose based on the skeleton information;
The registration method according to any one of appendices 10 to 13, wherein the pose setting step sets pose information based on the skeleton information and the used movable connection step information.
(Appendix 15)
The registration method described in Appendix 14, wherein the pose setting step displays the skeletal information on a display device, and superimposes visual information indicating the used movable connecting part information whose input has been received on the skeletal information displayed on the display device.
(Appendix 16)
16. The registration method according to any one of appendices 10 to 15, wherein the gesture creation information further includes control information regarding control of the pose information.
(Appendix 17)
Further, the method includes an operation information registration step,
The registration method according to any one of appendices 10 to 16, wherein the operation information registration step registers operation information related to an operation of an application in association with the registered gesture information.
(Appendix 18)
The method includes a registration process, a gesture image acquisition process, a recognition process, and an operation control process,
the registration step registers pose information set based on the skeleton information, the gesture information configured from one or more pieces of pose information, and the operation information related to an operation of an application linked to the gesture information by a registration method according to supplementary note 17;
The gesture image acquiring step acquires a gesture image including one or more pose images including the skeleton information,
The recognizing step recognizes the registered gesture information from the gesture image,
The operation control step includes controlling an input device based on the operation information linked to the recognized gesture information to operate an application.
(Appendix 19)
A program for causing a computer to execute a procedure including a pose registration procedure and a gesture registration procedure:
The pose registration step includes a pose image acquisition step, a pose setting step, and a registration step,
The pose image acquisition step acquires a registration pose image including skeleton information;
the pose setting step sets pose information based on the skeleton information;
The registration step includes registering the set pose information,
the gesture registration step includes an input receiving step, a gesture setting step, and a registration step;
The input receiving step includes receiving an input of gesture creation information;
The gesture creation information is information regarding creation of a gesture, and includes information regarding pose information selected by a user from among one or more of the pose information;
the gesture setting step includes setting gesture information including one or more pieces of pose information based on the gesture creation information;
The registration step registers the set gesture information.
(Appendix 20)
The pose registration procedure further includes a pose request procedure and a recognition procedure;
the pose request step includes requesting capture of a test pose image including a target for authentication assuming a pose indicated by the pose information;
The pose image acquisition step acquires a test pose image including the skeleton information,
the recognition step recognizes the set pose information from the test pose image;
20. The program according to claim 19, wherein the registration step registers the pose information if the pose information is recognized.
(Appendix 21)
The gesture registration step further includes a gesture request step, a gesture image acquisition step, and a recognition step;
the gesture request step includes requesting capture of a test gesture image including the authentication target making a gesture indicated by the gesture information;
the gesture image acquisition step acquires a test gesture image composed of a plurality of the pose images including the skeleton information;
the step of recognizing the set gesture information from the test gesture image;
21. The program according to claim 19, wherein the registration step registers the gesture information if the gesture information is recognized.
(Appendix 22)
the gesture request step causes a display device to display the one or more pieces of pose information constituting the gesture information;
22. The program according to claim 21, wherein the recognition step displays, for each piece of pose information, visual information indicating whether or not the pose information has been recognized, superimposed on the pose information displayed on the display device.
(Appendix 23)
The pose registration step further includes an input reception step,
the input receiving step receives an input of use movable connection procedure information regarding a movable connection procedure to be used for a pose based on the skeletal information;
23. The program according to any one of appendices 19 to 22, wherein the pose setting procedure sets pose information based on the skeletal information and the used movable connection procedure information.
(Appendix 24)
The program described in Appendix 23, wherein the pose setting step displays the skeletal information on a display device, and displays visual information indicating the used movable connecting part information whose input has been received, superimposed on the skeletal information displayed on the display device.
(Appendix 25)
25. The program of claim 19, wherein the gesture creation information further includes control information regarding control of the pose information.
(Appendix 26)
Further, the method includes a procedure for registering operation information,
26. The program according to any one of appendices 19 to 25, wherein the operation information registration step registers operation information related to an operation of an application in association with the registered gesture information.
(Appendix 27)
A program for causing a computer to execute procedures including a registration procedure, a gesture image acquisition procedure, a recognition procedure, and an operation control procedure:
The registration step includes registering pose information set based on the skeleton information, the gesture information configured from one or more pieces of pose information, and the operation information related to an operation of an application linked to the gesture information by a program described in Supplementary Note 26;
The gesture image acquisition step acquires a gesture image including one or more pose images including the skeleton information;
the recognition step recognizes the registered gesture information from the gesture image;
The operation control procedure controls an input device to operate an application based on the operation information linked to the recognized gesture information.
(Appendix 28)
A computer-readable recording medium having a program according to any one of appendices 19 to 27 recorded thereon.

The present invention allows users to register their own poses and create gestures from those poses. Therefore, the present invention is useful when performing input operations using gestures.

1 Display 100 Registration device 10 Pose registration unit 11 Pose image acquisition unit 12 Pose setting unit 13 Pose request unit 14 Recognition unit 15 Registration unit 16 Input reception unit 20 Gesture registration unit 21 Input reception unit 22 Gesture setting unit 23 Gesture request unit 24 Gesture image acquisition unit 25 Recognition unit 26 Registration unit 30 Operation information registration unit 40 Imaging unit 200 Operation device 201 Registration unit 202 Gesture image acquisition unit 203 Recognition unit 204 Operation control unit 205 Imaging unit 101 Central processing unit 102 Memory 103 Bus 104 Storage device 105 Input device 106 Display device 107 Communication device 108 Imaging device

Claims (10)

A pose registration unit and a gesture registration unit are included,
the pose registration unit includes a pose image acquisition unit, a pose setting unit, and a registration unit;
the pose image acquisition unit acquires a registration pose image including skeleton information;
The pose setting unit sets pose information based on the skeleton information,
The registration unit registers the set pose information,
the gesture registration unit includes an input receiving unit, a gesture setting unit, and a registration unit;
the input receiving unit receives an input of gesture creation information;
The gesture creation information is information regarding creation of a gesture, and includes information regarding pose information selected by a user from among one or more of the pose information;
the gesture setting unit sets gesture information including one or more pieces of pose information based on the gesture creation information;
The registration unit registers the set gesture information. The pose registration unit further includes an input reception unit,
the input receiving unit receives input of information on a movable link to be used for a pose based on the skeletal information;
The pose setting unit sets pose information based on the skeleton information and the used movable connecting part information.
The registration device according to claim 1. Further, an operation information registration unit is included,
The registration device according to claim 1 , wherein the operation information registration unit registers operation information relating to an operation of an application in association with the registered gesture information. A registration unit, a gesture image acquisition unit, a recognition unit, and an operation control unit are included.
the registration unit registers, by using a registration device according to claim 3, pose information set based on the skeletal information, the gesture information configured from one or more pieces of pose information, and the operation information related to an operation of an application linked to the gesture information;
the gesture image acquisition unit acquires a gesture image composed of one or more pose images including the skeleton information;
The recognition unit recognizes the registered gesture information from the gesture image,
The operation control unit controls an input device to operate an application based on the operation information linked to the recognized gesture information. The method includes a pose registration step and a gesture registration step,
The pose registration step includes a pose image acquisition step, a pose setting step, and a registration step,
The pose image acquisition step acquires a registration pose image including skeleton information,
The pose setting step sets pose information based on the skeleton information,
The registration step includes registering the set pose information,
The gesture registration step includes an input receiving step, a gesture setting step, and a registration step,
The input receiving step receives an input of gesture creation information;
The gesture creation information is information regarding creation of a gesture, and includes information regarding pose information selected by a user from among one or more of the pose information;
The gesture setting step sets gesture information including one or more pieces of pose information based on the gesture creation information,
The registration step registers the set gesture information. Further, the method includes an operation information registration step,
The registration method according to claim 5 , wherein the operation information registration step registers operation information relating to an operation of an application in association with the registered gesture information. The method includes a registration process, a gesture image acquisition process, a recognition process, and an operation control process,
The registration step includes registering pose information set based on the skeleton information, the gesture information configured from one or more pieces of pose information, and the operation information related to an operation of an application linked to the gesture information, by a registration method according to claim 6,
The gesture image acquiring step acquires a gesture image including one or more pose images including the skeleton information,
The recognizing step recognizes the registered gesture information from the gesture image,
The operation control step includes controlling an input device based on the operation information linked to the recognized gesture information to operate an application. A program for causing a computer to execute a procedure including a pose registration procedure and a gesture registration procedure:
The pose registration step includes a pose image acquisition step, a pose setting step, and a registration step,
The pose image acquisition step acquires a registration pose image including skeleton information;
the pose setting step sets pose information based on the skeleton information;
The registration step includes registering the set pose information,
the gesture registration step includes an input receiving step, a gesture setting step, and a registration step;
The input receiving step includes receiving an input of gesture creation information;
The gesture creation information is information regarding creation of a gesture, and includes information regarding pose information selected by a user from among one or more of the pose information;
the gesture setting step includes setting gesture information including one or more pieces of pose information based on the gesture creation information;
The registration step registers the set gesture information. Further, the method includes a procedure for registering operation information,
The program according to claim 8 , wherein the operation information registration step registers operation information relating to an operation of an application in association with the registered gesture information. A program for causing a computer to execute procedures including a registration procedure, a gesture image acquisition procedure, a recognition procedure, and an operation control procedure:
The registration step includes registering pose information set based on the skeleton information, the gesture information configured from one or more pieces of the pose information, and the operation information related to an operation of an application linked to the gesture information by a program according to claim 9 ;
The gesture image acquisition step acquires a gesture image including one or more pose images including the skeleton information;
the recognition step recognizes the registered gesture information from the gesture image;
The operation control procedure controls an input device to operate an application based on the operation information linked to the recognized gesture information.

Images