JP7158261B2 - Information processing device, control program, recording medium - Google Patents

Description

The present invention relates to an information processing apparatus and a gesture operation detection method.

Various techniques have been developed for capturing images of gesture operations performed by an operator with a camera and operating robots according to the gesture operations.

For example, in a robot system disclosed in Patent Document 1, an operator's gesture is detected based on imaging information of an operator, a robot control command associated with the detected gesture is identified, and the robot control command is detected. A notification image for notifying the operator of is displayed on the display unit.

JP 2014-104527 A

The robot system described in Patent Literature 1 identifies a command associated with a gesture operation based on the captured motion of the operator. However, when the operator is too close to or too far from the camera, there arises a problem that the operator's gesture operation cannot be captured appropriately and the gesture operation cannot be detected with high accuracy.

In addition, when a plurality of operators can each perform a gesture operation, it is desirable to preferentially detect a gesture operation with high urgency and importance with high accuracy.

An object of one aspect of the present invention is to realize an information processing device, a control method for the information processing device, and the like that can accurately detect a gesture operation having a high priority.

To solve the above problems, an information processing apparatus according to one aspect of the present invention is an information processing apparatus that detects a gesture operation, comprising at least one imaging device, at least one moving mechanism, and at least one moving mechanism. and a control device, wherein when a plurality of operators each perform a gesture operation, the control device controls at least the operator performing the gesture operation with the highest priority by adjusting the body of the operator when performing the gesture operation. estimating a target area in which a predetermined part of the movement is performed, and moving the imaging device to a position where the target area can be imaged and the predetermined part can be imaged at a predetermined resolution or higher A mechanism is controlled, the gesture operation is detected based on the motion of the predetermined part imaged by the imaging device from the position, and a higher priority is given based on a preset priority of the gesture operation. Identify gesture operations.

Further, in order to solve the above-described problems, a control method for an information processing apparatus according to an aspect of the present invention provides an operation for performing at least a gesture operation with the highest priority when a plurality of operators each perform a gesture operation. A control method for an information processing device that detects a gesture operation of a person, comprising: an estimation step of estimating a target area in which a predetermined part of the body of the operator moves when the gesture operation is performed; Based on a moving step of moving a target region to a position where an image can be captured and the predetermined part can be imaged at a predetermined resolution or higher, and an operation of the predetermined part imaged by the imaging device from the position. A detection step of detecting the gesture operation, and an identification step of identifying a gesture operation with a higher priority based on the preset priority of the gesture operation.

According to one aspect of the present invention, it is possible to realize an information processing device, a control method for the information processing device, and the like that can accurately detect a gesture operation having a high priority.

1 is a block diagram showing an example of a schematic configuration of an information processing apparatus according to Embodiment 1 of the present invention; FIG. FIG. 4 is a conceptual diagram illustrating an example of an operation in which an information processing device detects a gesture operation of an operator; It is a block diagram which shows an example of the hardware constitutions of an information processing apparatus. 7 is a flow chart showing an example of a flow of processing in which an information processing device identifies an operator with a high priority and detects a gesture operation of the operator; It is a figure which shows an example of the data structure of a priority list. 6 is a flowchart showing an example of a flow of processing for detecting a gesture operation by an information processing device; (a) and (b) are diagrams for explaining an example in which the information processing apparatus moves to detect a gesture operation, and (c) is an example of a target area for the information processing apparatus to detect a gesture operation. It is a figure explaining. FIG. 3 is a block diagram showing another example of the schematic configuration of the information processing apparatus; 9 is a flowchart showing another example of the flow of processing for the information processing device to detect a gesture operation; It is a figure explaining an example of operation|movement of an information processing apparatus. It is a figure explaining another example of operation|movement of an information processing apparatus. FIG. 3 is a block diagram showing an example of a schematic configuration of an information processing apparatus according to Embodiment 2 of the present invention; 7 is a flow chart showing an example of the flow of processing by an information processing device to identify a gesture operation with a high priority among detected gesture operations. It is a figure which shows an example of the data structure of a priority list. FIG. 5 is a block diagram showing an example of a schematic configuration of an information processing apparatus according to Embodiment 3 of the present invention; It is a figure which shows an example of the data structure of a priority list.

[Embodiment 1]
An embodiment of the present invention will be described in detail below.

(Overview of information processing device 1)
The information processing apparatus 1 according to this embodiment has a function of detecting a gesture operation by an operator. In a work site where a plurality of operators who are authorized to perform gesture operations work, the information processing apparatus 1 may detect gesture operations by a plurality of operators. In such a case, it is desirable that the information processing apparatus 1 detect a gesture operation that is more important and more urgent (that is, has a higher priority).

Therefore, the information processing apparatus 1 has a function of identifying a gesture operation with a high priority based on preset priorities of gesture operations. Specifically, the information processing apparatus 1 detects a gesture operation by a high-priority operator as a high-priority gesture operation.

The operator's priority may be determined in advance for each operator. For example, operators who have the authority to perform gesture operations on the information processing apparatus 1 are registered in advance in the priority list 23, and a unique priority is given to each operator for each appearance feature (for example, the operator's face, etc.). degree may be specified. The information processing apparatus 1 can distinguish between the operator with the highest priority and other operators based on the appearance features of the captured operator.

(Detection of gesture operation by information processing device 1)
The information processing device 1 includes at least one camera 31 (imaging device) that captures an image of the operator P, at least one moving mechanism 35, and at least one control unit 10 (control device) (see FIG. 1 reference). The information processing apparatus 1 has a function of autonomously moving to a position suitable for capturing an image of the action of the operator P by the camera 31 and detecting the gesture operation of the operator P. FIG.

The information processing apparatus 1 is not limited to these, but for example, (1) a robot capable of autonomous running or independent walking, which includes tires, caterpillars, and/or walking legs as a moving mechanism 35, (2) ) a robot capable of autonomous flight provided with a moving mechanism 35 such as a propeller; and (3) a robot capable of autonomous navigation provided with a moving mechanism 35 such as a screw. Alternatively, the information processing apparatus 1 may be a robot including a plurality of the moving mechanisms 35 described in (1) to (3) above.

First, a method of detecting a gesture operation will be described with reference to FIG. FIG. 2 is a conceptual diagram illustrating an example of the operation of the information processing device 1 detecting a gesture operation by the operator P. As shown in FIG.

The information processing apparatus 1 equipped with the camera 31 captures an image of the operator P using the camera 31 . However, if the distance between the camera 31 and the operator P is too short, the region R1 captured by the camera 31 includes a region where a predetermined part of the body of the operator P (for example, the hand P1) moves (hereinafter referred to as (referred to as the region of interest) is included. In addition, below, although the case where the predetermined site|part is the operator's P hand P1 is mentioned as an example and demonstrated, it is not limited to this. For example, the predetermined parts may be the operator P's legs and face (eg, eyes and mouth).

The size of the region R1 depends on the angle of view of the camera 31 and the distance between the camera 31 and the operator P. That is, the closer the camera 31 and the operator P are, the smaller the region R1 is, and the further the camera 31 and the operator P are apart, the larger the region R1 is. On the other hand, if the camera 31 is farther away from the operator P than the appropriate distance, the resolution of the area R1 imaged by the camera 31 is lowered. If the resolution of the region R1 is lower than the predetermined resolution, the gesture operation performed by moving a predetermined part in the gesture operation cannot be accurately detected.

Therefore, as shown in FIG. 2, the information processing apparatus 1 estimates a target region R2 where the hand P1 of the operator P moves in order to perform imaging for detecting a gesture operation by the operator P, moves autonomously to a position where the target region R2 can be imaged by and the hand P1 of the operator P can be imaged at a predetermined resolution or higher. The movement of the hand P1 of the operator P differs for each gesture operation. Therefore, the predetermined resolution may be any resolution that allows the movement of the operator P's hand P1 to be determined.

In FIG. 2, the information processing apparatus 1, which has been at a position close to the operator P, autonomously moves away from the operator P, and steadily detects the gesture operation by the operator P as an example. explained. On the contrary, even if the information processing device 1 is located far away from the operator P, the same is true. That is, when the information processing device 1 exists at a position far away from the operator P, the information processing device 1 autonomously moves toward the operator P and detects the operator's gesture operation.

The information processing device 1 that has detected a gesture operation by the operator P identifies a control command associated with the detected gesture operation. The information processing apparatus 1 may be configured to operate according to the specified command, or may be configured to transmit the specified control command to a work device other than the information processing apparatus 1. may

In this way, the information processing apparatus 1 autonomously moves to a position where the distance between itself and the operator P is within an appropriate distance range for detecting the operator's gesture operation, and the operator P is imaged. Thereby, the information processing apparatus 1 can detect the gesture operation of the operator P with high accuracy.

(Configuration of information processing device 1)
Next, the configuration of the information processing device 1 will be described with reference to FIGS. 1, 3, and 5. FIG. FIG. 1 is a block diagram showing an example of a schematic configuration of an information processing device 1. As shown in FIG. FIG. 3 is a block diagram showing an example of the hardware configuration of the information processing device 1. As shown in FIG.

The information processing device 1 includes a control unit 10 that controls and controls each unit of the information processing device 1 and a storage unit 20 that stores various data used by the information processing device 1 . The information processing device 1 further includes a camera 31 , a distance measuring sensor 32 , a moving mechanism control section 33 , a moving mechanism 35 and a communication section 36 .

<Camera 31>
The camera 31 is mounted on the information processing device 1 and is capable of imaging the surroundings of the information processing device 1 . More specifically, camera 31 is a digital camera or digital video. Note that the camera 31 may include a lens with a fixed focal length, or a lens with a variable magnification (for example, a zoom lens). The camera 31 may be a so-called PTZ camera that has pan, tilt and zoom functions.

The information processing apparatus 1 may be configured to change the orientation of the camera 31 by rotating itself, or may be configured to further include an orientation adjustment unit (not shown) for adjusting the orientation of the camera 31. may

<Range sensor 32>
The distance measuring sensor 32 is a sensor for measuring the distance between a distance measuring object (for example, the operator P, etc.) and the information processing apparatus 1 . The ranging sensor 32 may be, for example, a laser sensor that uses infrared rays.

Note that the information processing apparatus 1 may be configured to estimate the distance between the camera 31 and the operator P, for example, based on the captured image captured by the camera 31. In this case, the distance measuring sensor 32 there is no need to prepare More specifically, in the information processing apparatus 1, the control unit 10 uses the function of the camera control unit 11 to adjust the focus of the camera 31, and the control unit 10 estimates the distance from the camera 31 to the operator P whose image is captured. may be 1 and 3 show a configuration example of an information processing device 1 including a distance measuring sensor 32. FIG.

<Storage unit 20>
The storage unit 20 in FIG. 1 corresponds to the ROM 201 in FIG. The ROM in FIG. 3 stores an operation program for controlling the operation of each unit provided in the information processing apparatus 1 . On the other hand, the RAM 202 in FIG. 3 is a working memory for temporarily storing data such as captured images used for the operation of the information processing apparatus 1 .

The storage unit 20 in FIG. 1 includes a camera specification 21, a gesture operation list 22, and a priority list 23.

The camera specification 21 stores information on the performance of the camera 31, such as the angle of view and resolution. Note that when the camera 31 is replaceable, the camera specifications 21 store information about the performance of all the cameras 31 that can be mounted on the information processing device 1, and the control unit 10 is mounted on the information processing device 1. The configuration may be such that information about the performance of the camera 31 that is present is acquired.

The gesture operation list 22 is a list of control commands associated with gesture operations to be detected by the information processing device 1 . More specifically, the gesture operation list 22 is a list that defines correspondence relationships between motion patterns of the hand P1 of the operator P and control commands associated with each motion pattern.

The priority list 23 is data that holds the correspondence between the appearance features of each operator and the priority. For example, as shown in FIG. 5, each operator's face can be used as the appearance feature. Alternatively, the color of the helmet and clothes worn by each operator may be used as the appearance features.

As shown in FIG. 5, the priority of each operator may be assigned to each operator (that is, the order of priority), but is not limited to this. For example, each operator may be classified according to affiliation, rank, and division of roles, and a priority order may be assigned to each affiliation, rank, and division of roles. However, if there are a plurality of operators belonging to the same category, it is desirable to assign a priority order to each operator.

<Control unit 10>
The controller 10 in FIG. 1 corresponds to the CPU 100 in FIG.

The control unit 10 includes a camera control unit 11 , an object detection unit 12 , a person determination unit 13 , a priority determination unit 14 , an area estimation unit 15 , a position determination unit 16 and an operation detection unit 17 .

The camera control unit 11 controls the start of imaging by the camera 31, focus adjustment during imaging, exposure adjustment, imaging stop, and the like. In addition, the camera control unit 11 controls the imaging direction of the camera 31 and controls the camera 31 so as to image the surroundings in order to search for the operator P. FIG. Furthermore, the camera control unit 11 acquires data of the captured image captured by the camera 31 .

The object detection unit 12 applies a known image recognition technique to check whether or not a person is shown in the captured image acquired by the camera control unit 11 . When a person is captured, the object detection unit 12 extracts an area corresponding to the person in the captured image. The object detection unit 12 may extract at once regions corresponding to all persons in which a plurality of persons are shown in the captured image acquired by the camera control unit 11, or may individually extract regions corresponding to each person. You may

The person determination unit 13 performs a process of detecting an operator in the area extracted by the object detection unit 12 according to appearance features. For example, the person determination unit 13 may determine whether or not a registered face (see FIG. 5) in the priority list 23 has been detected. Alternatively, the person determination unit 13 determines whether the operator has been detected according to the appearance features such as the hat, shoes, and clothes worn by the person appearing in the area extracted by the object detection unit 12. may be determined. In this case, the appearance features are colors, patterns, and the like.

The priority determination unit 14 refers to the priority list 23 and identifies the operator with the highest priority among the persons determined to be the operators by the person determination unit 13 as the operator P whose gesture operation should be detected. do.

The region estimation unit 15 estimates a target region R2 in which the hand P1 of the operator P moves when the operator P performs a gesture operation, based on the region corresponding to the operator P identified by the priority determination unit 14. do. Note that the region estimation unit 15 determines the central axis L (see FIG. 7) of the body of the operator P based on the extracted region corresponding to the operator P, and estimates the inclination of the body of the operator P. may A method of estimating the target region R2 by the region estimation unit 15 will be described later.

The position determination unit 16 determines whether or not the distance between the camera 31 and the operator P, which is measured by the distance measuring sensor 32, is within a distance range suitable for detecting the gesture operation of the operator P. . That is, it is determined whether or not the camera 31 is present at a suitable position for detecting the gesture operation of the operator P. A position suitable for detecting the gesture operation of the operator P is a position where the target region R2 can be imaged by the camera 31 and the hand P1 of the operator P can be imaged with a predetermined resolution or higher. A distance range between the camera 31 and the operator P suitable for detecting the gesture operation of the operator P is calculated based on the size of the target region R2, the resolution of the camera 31, and the angle of view of the camera 31. be done.

If the camera 31 is presently located at a position suitable for detecting the gesture operation of the operator P, the position determination unit 16 performs camera control so that the camera 31 captures the motion of the hand P1 of the operator P. Instruct part 11. On the other hand, if the camera 31 is not currently at a position suitable for detecting the gesture operation of the operator P, the position determination unit 16 instructs the movement mechanism control unit 33 to Instruct to move to

The operation detection unit 17 detects the motion of the operator P's hand P1 captured at a position suitable for detecting the gesture operation of the operator P, and determines whether the detected motion is included in the gesture operation list 22. confirm whether or not If the detected motion is included in gesture operation list 22 , operation detection unit 17 determines a control command associated with the detected motion of hand P<b>1 based on gesture operation list 22 . When detecting a gesture operation, the operation detection unit 17 instructs the camera control unit 11 to change the magnification of the lens of the camera 31, and images the hand P1 of the operator P performing the gesture operation. You may be instructed to do so.

<Movement Mechanism Control Unit 33>
The moving mechanism control unit 33 is a logic circuit formed in an integrated circuit (IC chip) or the like for controlling start/stop of the moving mechanism 35 . 1 and 3, the control unit 10 (CPU 100 in FIG. 3) has the function of the moving mechanism control unit 33, and controls such as starting/stopping of the moving mechanism 35 are controlled by a software program. It may be a configuration that is realized by the instruction of

<Moving mechanism 35>
The moving mechanism 35 is a moving means for moving the information processing device 1, and includes tires, caterpillars, legs for walking, propellers, screws, etc., and one or a plurality of arbitrary power generators for moving these. contains. FIG. 1 exemplarily shows the moving mechanism 35 including a first motor 351, a second motor 352, a third motor 353, .

<Communication unit 36>
The communication unit 36 is for transmitting and receiving data between the information processing apparatus 1 and another work device (not shown). For example, the control command determined by the operation detection section 17 is transmitted to another work device via the communication section 36 .

Note that the information processing apparatus 1 illustrated in FIG. 1 shows a configuration example in which a control command associated with a gesture operation detected by the information processing apparatus 1 is transmitted to another work device via the communication unit 36. FIG. However, the present invention is not limited to this. For example, a processing execution unit (not shown) that executes predetermined processing according to a control command associated with a gesture operation detected by the information processing device 1 may be provided.

(Processing flow of information processing device 1)
Next, the flow of processing by the information processing apparatus 1 will be described with reference to FIG. FIG. 4 is a flowchart illustrating an example of a flow of processing in which an information processing device specifies an operator with a high priority and detects a gesture operation of the operator.

First, the camera control unit 11 captures an image of the surroundings of its own device with the camera 31 (step S11). Next, the object detection unit 12 extracts a region corresponding to a person from the captured image acquired by the camera control unit 11 . Then, the person determination unit 13 detects a person's face in the exposed area (step S12).

Next, the person determination unit 13 reads out the priority list 23 and determines whether or not each detected face has been registered in the priority list 23 (step S13). If a registered face is not detected (NO in step S13), the process returns to S11. If a registered face is detected (YES in step S13), the process proceeds to step S14.

Next, the priority determination unit 14 determines whether or not a plurality of registered faces (that is, a person whose face is registered as an operator) has been detected in step S13 (step S14). If the number of detected operators is plural (YES in step S14), the priority determination unit 14 refers to the priority list 23 to identify the priority of each detected operator, and the operator with the highest priority. A tall person is selected (step S15). Next, the priority determination unit 14 identifies the operator whose gesture operation should be detected based on the selection result described above (step S16: identification step). On the other hand, when one person is detected (NO in step S14), the priority determination unit 14 identifies the person as the operator P.

After that, a gesture operation is detected by each process described later (step S17). After that, the operation detection unit 17 reads out the gesture operation list 22, compares it with the detected gesture operation, and executes processing according to the detected gesture (step S18).

(Example of processing to detect gesture operation)
Next, an example of a process (step S17 in FIG. 4) of detecting a gesture operation by the information processing apparatus 1 will be described with reference to FIGS. 6 and 7. FIG. FIG. 6 is a flowchart showing an example of the flow of processing for detecting a gesture operation by the information processing device 1. As shown in FIG. (a) and (b) of FIG. 7 are diagrams for explaining an example in which the information processing device 1 moves to detect a gesture operation, and (c) of FIG. It is a figure explaining an example of the target area for detection.

First, the camera control unit 11 starts imaging the operator P with the camera 31 (step S1). Next, the target detection unit 12 applies a known image recognition technique to check whether or not at least part of the body of the operator P is shown in the captured image. As shown in (a) of FIG. 7, when the operator P is captured (that is, when the region R1 imaged by the camera 31 includes part of the operator P) (in step S2 YES), the object detection unit 12 extracts a region corresponding to the operator P in the captured image. If the operator P is not captured, or if the area corresponding to the operator P in the captured image cannot be extracted (NO in step S2), the process returns to step S1, and the camera control unit 11 causes the camera 31 to capture the surrounding area. Imaging and searching for the operator P are continued.

Next, the region estimation unit 15 estimates a target region R2 in which the hand P1 or P2 of the operator P moves when the operator P performs a gesture operation (step S3: estimation step).

The position determination unit 16 acquires the distance between the camera 31 and the operator P measured by the distance measuring sensor 32 (step S4), and the current distance between the camera 31 and the operator P is It is determined whether or not the distance range is suitable for imaging the region R2 (step S5). In (a) of FIG. 7, since the distance d1 between the camera 31 and the operator P is shorter than the distance suitable for imaging the target region R2, the hands P1 and P2 of the operator P cannot be imaged. This is an example of when not.

If the current distance between the camera 31 and the operator P is not within the distance range suitable for detecting a gesture operation, the position determination unit 16 instructs the movement mechanism control unit 33 to Instruct them to move to a suitable position. Accordingly, the information processing apparatus 1 moves to adjust the distance between the camera 31 and the operator P (step S6: moving step), returns to step S4, and adjusts the distance between the camera 31 and the operator P. to confirm. FIG. 7B shows that as a result of adjusting the distance, the current distance d2 between the camera 31 and the operator P is within the distance range suitable for imaging the target region R2, and the operator P It shows how the motions of hands P1 and P2 can be imaged.

Note that when the operator P has moved, the distance between the camera 31 and the operator P measured after step S6 may not yet be within the distance range suitable for detecting the gesture operation. Even in such a case, the information processing apparatus 1 can move following the moving operator P and detect the gesture operation of the operator P.

On the other hand, if the current distance between the camera 31 and the operator P is within the distance range suitable for detecting the gesture operation (YES in step S5), the position determination unit 16 causes the camera 31 to perform the operation. The camera control unit 11 is instructed to image the motion of the hand P1 of the person P (step S7).

Next, the operation detection unit 17 detects the gesture operation of the operator P imaged in step S7 (step S8: detection step).

Since the information processing apparatus 1 autonomously moves to a position where the distance between the camera 31 and the operator P is within an appropriate distance range for detecting the gesture operation of the operator P, the operation The gesture operation of the person P can be detected with high accuracy.

<Method for Estimating Target Region R2>
Here, a method of estimating the target region R2 will be described with reference to FIG. 7(c). (c) of FIG. 7 is a diagram illustrating an example of a target region R2 for detecting a gesture operation. In addition, in (c) of FIG. 7 , the case where the predetermined parts to be moved in the gesture operation are the hands P1 and P2 of the operator P will be described as an example, but the present invention is not limited to this. For example, it may be one hand (for example, P1) of the operator P, or the operator P's face, legs, or the like. That is, the estimated target region R2 can be appropriately changed in accordance with the motion part of the gesture operation to be detected by the information processing device 1 .

The region estimation unit 15 estimates the target region R2 based on general body shape statistical data. As the body type statistical data, for example, the "Japanese human body size database" issued by the Research Center for Human Engineering for Life (General Incorporated Association) can be applied. This "Japanese human body size database" contains statistical data such as shoulder width, hand length, upper arm length, and head width for general Japanese people.

The area estimation unit 15 determines the central axis L in the area where the operator P is imaged. Further, based on the ratio between the head width or shoulder width data and the upper arm length data in the human body size database of Japanese, the region estimation unit 15 calculates the width of the hand P1 when the operator P spreads both arms laterally. and P2 reach.

In this way, referring to a database such as the "Japanese Human Body Dimension Database", the region estimating unit 15 estimates the target region R2 that includes the distance m left and right from the central axis L of the operator P. can do.

Note that the target region R2 estimated by this method is strictly based on the human body dimensions of general Japanese people. There is a possibility that the tips of the hands P1 and P2 will not be captured. Therefore, it is desirable that the area estimation unit 15 estimates an area obtained by enlarging the vertical and horizontal widths of the area where the hands P1 and bP2 move by a certain ratio (for example, 5 to 10%) as the target area R2.

<Modification>
As described above, the information processing device 1 detects a gesture operation with a high priority through the process illustrated in FIG. 4 . However, the present invention is not limited to this. For example, the captured image captured by the camera 31 does not always include a person's face. Therefore, the priority of each person may be determined using appearance features such as the color of the helmet and clothes worn by the person.

Alternatively, the priority list 23 may define priority according to each operator's affiliation, class, role assignment, etc., instead of a unique priority for each person. If a plurality of operators with high priority are selected as a result of the classification, the operator P with the highest priority may be appropriately specified from among the selected operators.

When the information processing apparatus 1 takes an image of the operator on the ground from the sky, the operator P's face cannot always be imaged. For example, if the information processing device 1 is a drone, the higher the flying altitude, the more difficult it is for a person's face to be imaged. However, hats, helmets, armbands, clothing, and the like may be imaged. Therefore, in addition to a person's face, hats, helmets, armbands, clothing, and the like can be used as appearance features. For example, in step S12 of FIG. 4, the object detection unit 12 extracts an area that can be determined as a person from the photographed image using a known image recognition technique, and detects the color of the hat, the color of the clothing, or the pattern of the clothing. etc. may be detected. In this case, the priority determination unit 14 may refer to the priority list 23 and specify the operator P with the highest priority based on the extracted appearance features.

Further, the resolution required for the camera 31 may be lower for detecting the color of clothing than for detecting the operator's face. Therefore, if the information processing apparatus 1 is configured to identify an operator with a high priority according to the color of the hat, the color of the clothes, or the pattern of the clothes, the processing can be performed at a higher speed. The operator P can be identified even when the distance is long.

In addition, it may be specified by a combination of not only one appearance feature. In other words, two or more appearance features may be used as keys, and the priority may be specified by the combination of each key.

By adopting the above configuration, the information processing apparatus 1 can preferentially and accurately detect gesture operations with high urgency and importance even when a plurality of operators can perform gesture operations.

(Modification 1 of gesture operation detection processing)
The posture of the operator P is not limited to standing or sitting, but may be leaning against a wall or the like (a posture in which the body axis L3 is not parallel to the direction of gravity) or lying down. It is also conceivable to Therefore, the information processing apparatus 1 may be configured as shown in FIG. By adopting such a configuration, the information processing apparatus 1 can accurately detect the gesture operation of the operator P even when the body axis L3 of the operator P is tilted from the direction parallel to the gravity. It is possible.

<Modification of Configuration of Information Processing Apparatus 1>
FIG. 8 is a block diagram showing another example of the schematic configuration of the information processing device 1. As shown in FIG. The information processing apparatus 1 includes an orientation adjustment unit 39 which is a mechanism for controlling the imaging direction of the camera 31, and a gyro sensor 37 and an acceleration sensor 38 for monitoring the tilt of the camera 31 and the like. The information processing apparatus 1 also differs from the information processing apparatus 1 shown in FIG. 1 in that it includes a position/tilt determination unit 18 instead of the position determination unit 16 .

The orientation adjustment unit 39 is a mechanism for adjusting the imaging direction of the camera 31 . The orientation adjustment section 39 is controlled by the camera control section 11 . By actuating the orientation adjustment unit 39, the imaging direction of the camera 31 can be adjusted horizontally and vertically. For example, the orientation adjustment unit 39 adjusts the imaging direction of the camera 31 horizontally toward the operator P's one hand P1, or adjusts the imaging direction of the camera 31 downward to face the operator P in the supine position. It is possible to adjust. The orientation adjustment unit 39 may include a mechanism that rotates the camera 31 about the imaging direction L1 as a rotation axis.

The position/tilt determination unit 18 determines whether or not the distance between the camera 31 and the operator P, which is measured by the distance measuring sensor 32, is within a suitable distance range for detecting the gesture operation of the operator P. judge. Further, the position/inclination determination unit 18 detects the inclination of the body axis L3 of the operator P estimated by the region estimation unit 15, and the angle formed by the body axis L3 of the operator P and the imaging direction L1 of the camera 31 is It is determined whether or not it is within a predetermined range (eg, 60° to 120°). The body axis L3 is intended to be an axis passing through the centers of the operator P's head and torso.

When the angle formed by the body axis L3 of the operator P and the imaging direction L1 of the camera 31 is within a predetermined range, the position/inclination determination unit 18 detects the motion of the operator P or the hand P1 of the operator P by the camera 31. The camera control unit 11 is instructed to image the operation of . On the other hand, if the angle formed by the body axis L3 of the operator P and the imaging direction L1 of the camera 31 is not within the predetermined range, the position/tilt determination unit 18 instructs the camera control unit 11 to The direction of the camera 31 is adjusted so that the angle formed by the body axis L3 and the imaging direction L1 of the camera 31 is within a predetermined range.

When the position/tilt determination unit 18 detects the tilt of the body axis L3 of the operator P, the position/tilt determination unit 18 instructs the camera control unit 11 to It may be configured to instruct to rotate the camera 31 by an angle. Alternatively, the position/tilt determination unit 18 causes the operation detection unit 15 to rotate/rotate the captured image data by the same angle as the tilt of the body axis L3 of the operator P in the direction opposite to the tilt of the body axis L3 of the operator P. A configuration may be used in which an instruction is given to correct and detect a gesture operation.

By adopting such a configuration, the information processing apparatus 1 can accurately detect even a gesture operation performed by the operator P whose body axis L3 is not parallel to gravity.

Next, the processing flow of the information processing apparatus 1 that employs the configuration shown in FIG. 8 will be described with reference to FIGS. 9 and 10. FIG. FIG. 9 is a flowchart showing another example of the flow of processing for the information processing apparatus 1 to detect the gesture operation of the operator P, and FIG. It is a figure explaining the example which moves to. Since steps S1 to S5 are the same as in FIG. 4, the description thereof will not be repeated.

If the distance between the camera 31 and the operator P is not within a distance range suitable for detecting a gesture operation, the position/tilt determination unit 18 instructs the movement mechanism control unit 33 to Instruct them to move to a suitable position. Accordingly, the information processing device 1 moves to adjust the distance between the camera 31 and the operator P (step S6: moving step).

If the current distance between the camera 31 and the operator P is within the distance range suitable for detecting a gesture operation (YES in step S5), the position/tilt determination unit 18 moves the region estimation unit 15 (step S61), and it is determined whether or not the imaging direction of the camera 31 is suitable for detecting a gesture operation (step S62).

If the imaging direction of the camera 31 is not suitable for detecting a gesture operation (NO in step S62), the position/tilt determination unit 18 causes the camera control unit to match the orientation of the camera to the orientation of the body of the operator P. Point to 11. The camera control unit 11 controls the orientation adjustment unit 39 according to the instruction from the position/tilt determination unit 18, and determines that the angle formed by the body axis L3 of the operator P and the imaging direction L1 of the camera 31 is within a predetermined range. The direction of the camera 31 is adjusted so as to be (step S63), and the process returns to step S61. Note that when the predetermined part that moves in the gesture operation is only one hand P1, the area estimation unit 15 can also estimate the area in which the hand P1 moves as the target area R2. In this case, the imaging direction of the camera 31 may be adjusted so that the target region R2 can be imaged and the hand P1 can be imaged at a predetermined resolution or higher.

On the other hand, if the imaging direction of the camera 31 is suitable for detecting a gesture operation (YES in step S62), the position/inclination determination unit 18 causes the camera 31 to image the motion of the hand P1 of the operator P. The camera controller 11 is instructed to do so (step S7).

Next, the operation detection unit 15 detects the gesture operation of the operator P imaged in step S7 (step S8: detection step).

In (a) of FIG. 10, since the distance d3 between the camera 31 and the operator P in an inclined posture is shorter than the distance range suitable for imaging the target region R2, the operator P's hand A case where P1 and P2 cannot be imaged is illustrated. In this case, the information processing device 1a moves away from the operator P. FIG.

However, even if the distance between the camera 31 and the operator P becomes the distance d4, which is within the distance range suitable for imaging the target region R2, the body axis L3 of the operator P is tilted. Furthermore, there is a possibility that the angle B1 formed by the body axis L3 of the operator P and the imaging direction L1 of the camera 31 is not within a predetermined range, and the gesture operation by the motions of the hands P1 and P2 cannot be detected correctly. Therefore, as shown in FIG. 10B, the information processing apparatus 1 adjusts the angle between the body axis L3 of the operator P and the imaging direction L1 of the camera 31 to be an angle C1 within a predetermined range. adjust. As a result, the camera 31 can capture an image of the target region R2, and can capture an image of the hand P1 at a predetermined resolution or higher. In the example shown in (b) of FIG. 10 , the information processing apparatus 1 adjusts the imaging direction L1 of the camera 31 so that the operator P who is lying on his back and leaning backward can be imaged looking down.

(Modified Example 2 of Gesture Operation Detection Processing)
When the information processing device 1 autonomously moves in the air or in water, the information processing device 1 detects the gesture operation of the operator P present above or below. The information processing apparatus 1 (1) moves so that the distance between the camera 31 and the operator P is within a distance range suitable for detecting a gesture operation, and (2) adjusts the body axis L3 of the operator P. and the imaging direction L1 of the camera 31 is within a predetermined range. Therefore, the information processing apparatus 1 can detect the gesture operation of the operator P with high accuracy. This will be described with reference to FIG. 11 . FIG. 11 is a diagram illustrating another example in which the information processing apparatus 1 moves to detect the gesture operation of the operator P. FIG.

In (a) of FIG. 11, the distance d5 between the camera 31 of the information processing device 1a and the operator P is within a distance range suitable for imaging the motion of the operator P's hand P1. An angle B2 formed by the body axis L3 of the person P and the imaging direction L1 of the camera 31 is not within a predetermined range. Therefore, it is difficult to accurately detect the gesture operation by the action of the operator P's hand P1.

Therefore, as shown in FIG. 11B, the information processing apparatus 1 adjusts the angle between the body axis L3 of the operator P and the imaging direction L1 of the camera 31 to be an angle C2 within a predetermined range. adjust. In this case, the information processing apparatus 1 may be configured to change the flying altitude in addition to adjusting the imaging direction of the camera 31 . As a result, the camera 31 can capture an image of the target region R2, and can capture an image of the hand P1 at a predetermined resolution or higher. In the example shown in FIG. 11(b), the information processing apparatus 1 adjusts the imaging direction L1 of the camera 31 so as to look down on the operator P and lowers the flight altitude. Note that the distance d6 between the camera 31 of the information processing device 1 and the operator P is within a distance range suitable for imaging the motion of the operator P's hand P1, like the distance d5.

[Embodiment 2]
Other embodiments of the invention are described below. For convenience of description, members having the same functions as those of the members described in the above embodiments are denoted by the same reference numerals, and description thereof will not be repeated.

(Overview of information processing device 1a)
As with the information processing apparatus 1, the information processing apparatus 1a according to the present embodiment has a function of identifying a gesture operation with a high priority based on preset priorities of gesture operations. The information processing device 1 described above detects a gesture operation by an operator with a high priority as a gesture operation with a high priority. Identify the gesture operations with high priority among the .

The priority of the gesture operation may be determined in advance for each gesture operation. For example, gesture operations that can be performed by an operator who has the authority to perform gesture operations on the information processing device 1a are registered in the priority list 23a in advance, and a unique priority may be set for each gesture operation. Note that the priority of the gesture operation can be set according to, for example, the importance of acting in accordance with the gesture operation and the level of urgency.

(Configuration of information processing device 1a)
Next, the configuration of the information processing device 1a will be described with reference to FIG. FIG. 12 is a block diagram showing an example of a schematic configuration of the information processing device 1a.

The information processing device 1a includes a control unit 10a that controls all the components of the information processing device 1a, and a storage unit 20a that stores various data used by the information processing device 1a. The information processing apparatus 1 a further includes a camera 31 , a distance sensor 32 , a movement mechanism control section 33 , a movement mechanism 35 and a communication section 36 . Note that the configuration of the information processing device 1a is not limited to this. For example, the information processing device 1a, like the information processing device 1 shown in FIG. , an acceleration sensor 38 may be provided. Further, the information processing apparatus 1 a may further include a position/tilt determination section 18 instead of the position determination section 16 .

<Storage unit 20a>
The storage unit 20a includes a camera specification 21, a gesture operation list 22, and a priority list 23a.

The priority list 23a is data holding the priority of each gesture. Each gesture operation may be defined for each movement of a predetermined part of the operator's body (for example, hand) used for the gesture operation, as shown in FIG. 14, for example. The priority of each gesture operation is determined in advance by a list in which a unique priority is assigned to each gesture, as shown in FIG. 14, for example.

<Control unit 10a>
The control unit 10a corresponds to the CPU 100 in FIG.

The control unit 10 a includes a camera control unit 11 , an object detection unit 12 a, a priority determination unit 14 a, an area estimation unit 15 a, a position determination unit 16 a, and an operation detection unit 17 .

The object detection unit 12a applies a known image recognition technique to check whether or not the operator P appears in the captured image acquired by the camera control unit 11 . When the operator P is captured, the target detection unit 12a extracts an area corresponding to the operator P in the captured image. Note that the target detection unit 12a is different from the target detection unit 12 in that when a plurality of operators P are captured, regions corresponding to the respective operators P are individually extracted.

The region estimation unit 15a estimates a target region R2 for each region corresponding to the operator P extracted by the target detection unit 12a.

For each target region R2 estimated by the region estimating unit 15a, the position determining unit 16a determines that the distance between the camera 31 and each operator P measured by the distance measuring sensor 32 is used to detect the gesture operation. It is determined for each of the operators P whether or not they are within a distance range suitable for . That is, when the camera 31 is presently present at a position suitable for detecting the gesture operation of the operator P, the position determination unit 16a causes the camera 31 to capture an image of the motion of the hand P1 of the operator P. The camera control unit 11 is instructed. On the other hand, if the camera 31 is not currently at a position suitable for detecting the gesture operation of the operator P, the position determination unit 16 instructs the movement mechanism control unit 33 to Instruct to move to

The priority determination unit 14a refers to the priority list 23a to identify a gesture operation having a higher priority among the gesture operations detected by the operation detection unit 17. FIG. The priority determination unit 14a refers to the gesture operation list 22 and determines a control command associated with the gesture operation identified as the gesture operation with the highest priority.

(Processing flow of information processing device 1a)
Next, the processing flow of the information processing device 1a will be described with reference to FIG. FIG. 13 is a flowchart showing an example of the flow of processing for identifying a gesture operation with a high priority among the gesture operations detected by the information processing device 1a.

First, the camera control unit 11 captures an image of the surroundings of its own device with the camera 31 (step S21). Next, the target detection unit 12a applies a known image recognition technique to perform detection processing of the operator P on the captured image acquired by the camera control unit 11. FIG. When the operator P is captured (that is, when the region R1 captured by the camera 31 includes part of the operator P) (YES in step S22), the target detection unit 12 detects the captured image. A region corresponding to the operator P in the inside is extracted. If the operator P is not captured, or if the area corresponding to the operator P in the captured image cannot be extracted (NO in step S22), the process returns to step S21, and the camera control unit 11 causes the camera 31 to capture the surrounding area. Imaging and searching for the operator P are continued.

Next, the gesture operation performed by each operator P is detected by the processing shown in FIG. 6 or 9 (step S23). However, step S21 in FIG. 13 corresponds to step S1 in FIGS.

Next, when there is only one operator P performing the detected gesture operation (NO in step S24), the operation detection unit 17 outputs a control command associated with the detected gesture operation. decide. The priority determination unit 14a acquires the determined control command and executes processing according to the gesture operation (step S26). Note that the processing according to the gesture operation may include processing for transmitting a control command associated with the gesture operation to another work device.

If there are a plurality of operators P performing the detected gesture operation (YES in step S24), the operation detection unit 17 refers to the gesture operation list and performs control associated with the detected gesture operation. Decide on a command. Next, the priority determination unit refers to the priority list 23 a to obtain the priority of the gesture operation associated with the control command determined by the operation detection unit 17 . Next, the priority determination unit determines the gesture operation with the highest priority. (Step S25: specifying step). Thereafter, the control unit 10 executes processing according to the gesture operation (step S26).

By adopting the above configuration, the information processing apparatus 1b can preferentially and accurately detect gesture operations with high urgency and importance even when a plurality of operators can perform gesture operations.

[Embodiment 3]
Other embodiments of the invention are described below. For convenience of description, members having the same functions as those of the members described in the above embodiments are denoted by the same reference numerals, and description thereof will not be repeated.

(Overview of information processing device 1b)
As with the information processing apparatus 1, the information processing apparatus 1b according to the present embodiment has a function of identifying a gesture operation with a high priority based on preset priorities of gesture operations. The information processing device 1 described above detects a gesture operation by an operator with a high priority as a gesture operation with a high priority. Identify gesture operations with high priority among operations.

The priority of the gesture operation may be determined in advance for each gesture operation. For example, gesture operations that can be performed by an operator who has the authority to perform gesture operations on the information processing device 1b are registered in the priority list 23b in advance, and a unique priority is determined for each combination of the gesture operation and the operator. may have been Note that the priority for each combination can be set according to, for example, the importance of acting in accordance with the gesture operation and the level of urgency.

(Configuration of information processing device 1b)
Next, the configuration of the information processing device 1b will be described with reference to FIGS. 15 and 16. FIG. FIG. 15 is a block diagram showing an example of a schematic configuration of the information processing device 1b. FIG. 16 is a diagram showing an example of data holding priorities according to combinations of persons and gesture operations.

The information processing device 1b includes a control unit 10b that controls all the components of the information processing device 1b, and a storage unit 20a that stores various data used by the information processing device 1b. The information processing device 1 further includes a camera 31 , a distance sensor 32 , a moving mechanism control section 33 , a moving mechanism 35 and a communication section 36 .

Note that the configuration of the information processing device 1a is not limited to this. For example, the information processing device 1a, like the information processing device 1 shown in FIG. , an acceleration sensor 38 may be provided. Further, the information processing apparatus 1 a may further include a position/tilt determination section 18 instead of the position determination section 16 .

<Storage unit 20a>
The storage unit 20a of FIG. 15 includes a camera specification 21, a gesture operation list 22, and a priority list 23b.

The priority list 23b is different from the priority list 23 in that the priority list 23b holds the priority not only for each held person or gesture operation, but also for each combination of person and gesture operation. The priority of each combination, for example, as shown in FIG. 16, may be determined in advance by a list in which a unique priority is assigned to the combination of the priority of the operator and the priority of the gesture operation.

<Control unit 10b>
A control unit 10b in FIG. 15 corresponds to the CPU 100 in FIG.

The control unit 10 includes a camera control unit 11, an object detection unit 12, a priority determination unit 14b, an area estimation unit 15b, a position determination unit 16b, and an operation detection unit 17.

The priority determination unit 14b differs from the priority determination unit 14a in that priority is determined based on the output (determination result) from the person determination unit 13 and the output (detection result) from the operation detection unit 17. . Further, after the priority determination, instead of the operation detection unit 17, the priority determination unit 14b outputs to the communication unit 36b, which is different from the priority determination unit 14a.

The region estimation unit 15b differs from the region estimation unit 15a in that it performs processing based on the output (determination result) from the person determination unit 13. FIG.

By adopting the above configuration, the information processing apparatus 1b can preferentially and accurately detect gesture operations with high urgency and importance even when a plurality of operators can perform gesture operations.

[Example of realization by software]
The control blocks (especially the control units 10, 10a, and 10b) of the information processing apparatuses 1, 1a, and 1b may be realized by logic circuits (hardware) formed in integrated circuits (IC chips) or the like, or by software. may be realized by

In the latter case, the information processing apparatuses 1, 1a, and 1b are provided with computers that execute instructions of programs, which are software for realizing each function. This computer includes, for example, at least one processor (control device) and at least one computer-readable recording medium storing the program. In the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a "non-temporary tangible medium" such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. In addition, a RAM (Random Access Memory) for developing the above program may be further provided. Also, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. Note that one aspect of the present invention can also be implemented in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

The information processing device (1, 1a, 1b) according to each aspect of the present invention may be realized by a computer. A control program for an information processing apparatus that implements the above information processing apparatus on a computer, and a computer-readable recording medium recording the program are also included in the scope of the present invention.

The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

1, 1a, 1b information processing device 10, 10a, 10b control unit (control device)
31 camera (imaging device)
35 movement mechanism S3 estimation step S6 movement step S8 detection step S16, S25 identification step

Claims (9)

An information processing device that detects a gesture operation,
At least one imaging device, at least one movement mechanism, and at least one control device,
The control device is
When multiple operators each perform a gesture operation,
At least for the operator who performs the gesture operation with the highest priority,
estimating a target area in which a predetermined part of the operator's body moves when performing a gesture operation;
controlling the movement mechanism so as to move the imaging device to a position where the target region can be imaged and the predetermined site can be imaged at a predetermined resolution or higher;
An information processing apparatus, wherein the gesture operation is detected based on the motion of the predetermined part imaged by the imaging device from the position. The control device is
Determining the priority of each of the plurality of operators based on the appearance characteristics of each operator,
2. The information processing apparatus according to claim 1, wherein one of the plurality of operators is specified as the operator performing the gesture operation with the highest priority. The control device is
2. The information processing apparatus according to claim 1, wherein a gesture operation having the highest priority among the gesture operations performed by the plurality of operators is specified. The control device is
2. The information processing apparatus according to claim 1, wherein a gesture operation having a higher priority is specified based on the priority for each operator and the priority for each gesture operation. 5. The information processing according to any one of claims 1 to 4, wherein the control device images the predetermined part using at least one function of pan, tilt, and zoom of the imaging device. Device. further comprising an orientation adjustment unit that adjusts an imaging direction in which the imaging device images according to the inclination of the operator's body,
The control device is
2. The direction adjusting unit is controlled so as to be in the imaging direction in which the target area can be imaged and the movement of the predetermined part can be imaged at the predetermined resolution or higher. 6. The information processing apparatus according to any one of 5. The control device is
3. The orientation adjusting unit is controlled so that an angle formed between a body axis passing through the centers of the operator's head and torso and an imaging direction of the imaging device is within a predetermined range. 7. The information processing device according to 6. A control program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 7, the control program for causing the computer to function as the control apparatus. A computer-readable recording medium recording the control program according to claim 8 .

Images