JP5720231B2 - Image processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly, to an image processing apparatus, method, and program that can easily and accurately realize a non-contact input operation by a player on a game table. About.

  A game table that has been widely used is provided with a push button switch as a selection device. When a player plays on the game table, he or she operates the push button switch to move the symbols and the like displayed on the display unit according to various effect images displayed on the display unit, or By making them select, the effect of the game stand is enhanced (see Patent Documents 1 and 2).

JP 2004-24324 A JP 2005-168560 A

  However, it is known that a conventional push button switch is more likely to be hit by a player when a big hit or the like does not occur easily during the game than it is used for its original operation purpose. ing. For this reason, when such a pushbutton switch is provided, durability and resistance to such a rough operation as hitting and countermeasures against disturbance stresses such as static electricity are required, so that the development and cost thereof are large.

  The present invention has been made in view of such circumstances, and in particular, by using an image, it is possible to realize a non-contact, easy operation and accurate input device. .

An image processing apparatus according to an aspect of the present invention is an image processing apparatus that receives input information of a player based on a hand shape of a player's hand included in an image, and includes coordinates of a fingertip included in the image. Fingertip coordinate acquisition means for acquiring a position, area pattern recognition means for recognizing a pattern of an area to which the fingertip coordinates acquired by the fingertip coordinate acquisition means belong as an area pattern when dividing the image into a plurality of areas; Corresponding to an area pattern that matches the area pattern recognized by the area pattern recognition means, out of the area patterns stored in the storage means, and storing means that stores the area pattern of the fingertip coordinate position in association with the hand shape The hand shape stored with the hand shape is estimated as the hand shape in the image, and the estimation result is determined by the hand shape. Anda estimating means for outputting as the input information, the estimation unit area pattern recognized by the area pattern recognition means, of the hand shape included in the storage means in association with the predetermined hand shape memory If the input pattern matches the area pattern to be input, it is assumed that the supply of input information by the hand shape has been started, and thereafter, the area pattern recognized by the area pattern recognition means includes all the hands included in the storage means. When it matches one of the area patterns stored in association with the shape, it is estimated that the hand shape is stored in association with the matched area pattern, and the hand shape as the estimation result is output as input information. To do .

  The estimation means has a specific hand shape when the area pattern recognized by the area pattern recognition means does not match the area pattern stored in association with all the hand shapes included in the storage means. And the hand shape as the estimation result can be output as input information.

An image processing method according to one aspect of the present invention is an image processing apparatus that accepts input information of a player based on a hand shape of the player's hand included in an image, the coordinates of a fingertip included in the image. Fingertip coordinate acquisition means for acquiring a position, area pattern recognition means for recognizing a pattern of an area to which the fingertip coordinates acquired by the fingertip coordinate acquisition means belong as an area pattern when dividing the image into a plurality of areas; Corresponding to an area pattern that matches the area pattern recognized by the area pattern recognition means, out of the area patterns stored in the storage means, and storing means that stores the area pattern of the fingertip coordinate position in association with the hand shape The hand shape stored with the hand shape is estimated as the hand shape in the image, and the estimation result is determined by the hand shape. Anda estimating means for outputting as the input information, the estimation unit area pattern recognized by the area pattern recognition means, of the hand shape included in the storage means in association with the predetermined hand shape memory If the input pattern matches the area pattern to be input, it is assumed that the supply of input information by the hand shape has been started, and thereafter, the area pattern recognized by the area pattern recognition means includes all the hands included in the storage means. When it matches one of the area patterns stored in association with the shape, it is estimated that the hand shape is stored in association with the matched area pattern, and the hand shape as the estimation result is output as input information. an image processing method for an image processing apparatus which, in the fingertip coordinate acquisition means, the seat of the fingertip of the hand is contained in the image When a fingertip coordinate acquisition step for acquiring a position and the area pattern recognition unit divides the image into a plurality of areas, the pattern of the area to which the fingertip coordinates acquired by the processing of the fingertip coordinate acquisition step belongs is used as an area pattern. Among the area patterns stored in the storage means in the area pattern recognition step to be recognized and the estimation means, the area pattern is stored in association with the area pattern that matches the area pattern recognized by the processing of the area pattern recognition step. An estimation step of estimating a hand shape as a hand shape in the image and outputting an estimation result as input information based on the hand shape.

A program according to one aspect of the present invention is an image processing apparatus that receives input information of a player based on a hand shape of the player's hand included in an image, and the coordinate position of a fingertip of the hand included in the image is determined. Fingertip coordinate acquisition means for acquiring, area pattern recognition means for recognizing a pattern of an area to which the fingertip coordinates acquired by the fingertip coordinate acquisition means belong when dividing the image into a plurality of areas, and the hand shape A memory unit that stores an area pattern of the fingertip coordinate position in association with the area pattern, and an area pattern stored in the storage unit that is associated with an area pattern that matches the area pattern recognized by the area pattern recognition unit. The stored hand shape is estimated as the hand shape in the image, and the estimation result depends on the hand shape. Including an estimation unit for outputting as the force information, wherein the estimating means, the area pattern that is recognized by the area pattern recognition means, of the hand shape included in the storage means in association with the predetermined hand shape memory If the input pattern matches the area pattern to be input, it is assumed that the supply of input information by the hand shape has been started, and thereafter, the area pattern recognized by the area pattern recognition means includes all the hands included in the storage means. When it matches one of the area patterns stored in association with the shape, it is estimated that the hand shape is stored in association with the matched area pattern, and the hand shape as the estimation result is output as input information. a computer for controlling an image processing apparatus which, in the fingertip coordinate acquisition means, the seat of the fingertip of the hand is contained in the image When a fingertip coordinate acquisition step for acquiring a position and the area pattern recognition unit divides the image into a plurality of areas, the pattern of the area to which the fingertip coordinates acquired by the processing of the fingertip coordinate acquisition step belongs is used as an area pattern. Among the area patterns stored in the storage means in the area pattern recognition step to be recognized and the estimation means, the area pattern is stored in association with the area pattern that matches the area pattern recognized by the processing of the area pattern recognition step. An estimation step of estimating a hand shape as a hand shape in the image and outputting an estimation result as input information based on the hand shape.

  In one aspect of the present invention, the image processing apparatus accepts input information of the player based on the hand shape of the player's hand included in the image, and the coordinate position of the fingertip of the hand included in the image is acquired. When the image is divided into a plurality of areas, the pattern of the area to which the acquired fingertip coordinates belong is recognized as an area pattern, and the area pattern of the fingertip coordinate position is stored and stored in association with the hand shape. The hand shape stored in association with the area pattern that matches the recognized area pattern is estimated as the hand shape in the image, and the estimation result is input by the hand shape. Output as information.

  In the image processing device of one aspect of the present invention, the image processing device accepts input information of the player by the hand shape of the player's hand included in the image, and the coordinates of the fingertip of the hand included in the image The fingertip coordinate acquisition means for acquiring the position is, for example, a fingertip position calculation unit, and when dividing the image into a plurality of areas, the pattern of the area to which the fingertip coordinates acquired by the fingertip coordinate acquisition means belong is an area pattern. The area pattern recognition means that recognizes as, for example, a fingertip position area recognition section, and the storage means that stores the area pattern of the fingertip coordinate position in association with the hand shape is, for example, an area pattern storage section, Of the area patterns stored in the storage means, the area patterns corresponding to the area patterns recognized by the recognition means are associated. A hand shape stored Te is estimated as being the hand shape of the hand in the image, and the estimating means for outputting an estimation result as the input information by hand shape, e.g., a estimator.

  That is, when an image is divided into a plurality of areas, the area pattern detected as the fingertip coordinate position shown in the image matches the area pattern of the fingertip coordinate position stored in advance in association with the hand shape. When it does, it is estimated that it is a hand shape registered in association. For this reason, the player can perform an input operation only by presenting it in the imaging area by changing the hand shape in various ways, and thus it is possible to realize a non-contact and easy input operation. Become.

  According to the present invention, it is possible for a player to easily perform an input operation in a non-contact manner with respect to a game table.

It is an external view explaining the structural example of the game stand to which the image processing apparatus of this invention is applied. It is a block diagram which shows the structural example of a game stand. It is a figure which shows the structural example of a camera module. It is a block diagram which shows the structural example for implement | achieving the function of a camera module. It is a flowchart explaining an effect image display process. It is a flowchart explaining a fingertip tracking process. It is a figure explaining a fingertip tracking process. It is a flowchart explaining a hand shape estimation process. It is a figure explaining a hand shape estimation process. It is a flowchart explaining a rotation detection process. It is a figure explaining a rotation detection process. It is a figure explaining a rotation detection process. It is a flowchart explaining an area correction process. And FIG. 11 is a diagram illustrating a configuration example of a general-purpose computer.

  Embodiments of the present invention will be described below. Correspondences between the configuration requirements of the present invention and the embodiments described in the detailed description of the present invention are exemplified as follows. This description is to confirm that the embodiments supporting the present invention are described in the detailed description of the invention. Accordingly, although there are embodiments that are described in the detailed description of the invention but are not described here as embodiments corresponding to the constituent elements of the present invention, It does not mean that the embodiment does not correspond to the configuration requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  That is, an image processing apparatus according to one aspect of the present invention is an image processing apparatus that receives input information of a player based on a hand shape of the player's hand included in an image, and the fingertip of the hand included in the image. The fingertip coordinate acquisition means (for example, the fingertip position calculation unit 63 in FIG. 4) and the area of the area to which the fingertip coordinates acquired by the fingertip coordinate acquisition means belong when the image is divided into a plurality of areas. Area pattern recognition means (for example, fingertip position area recognition unit 101 in FIG. 4) that recognizes the pattern as an area pattern, and storage means (for example, in FIG. 4) that stores the area pattern of the fingertip coordinate position in association with the hand shape. Of the area patterns stored in the area pattern storage unit 102) and the storage means, an area that matches the area pattern recognized by the recognition means. Estimating means for estimating the hand shape stored in association with the pattern as the hand shape in the image and outputting the estimation result as input information based on the hand shape (for example, the estimation unit in FIG. 4). 103).

  An image processing method according to one aspect of the present invention is an image processing apparatus that accepts input information of a player based on a hand shape of the player's hand included in an image, the coordinates of a fingertip included in the image. Fingertip coordinate acquisition means for acquiring a position, area pattern recognition means for recognizing a pattern of an area to which the fingertip coordinates acquired by the fingertip coordinate acquisition means belong as an area pattern when dividing the image into a plurality of areas; Corresponding to an area pattern that matches the area pattern recognized by the area pattern recognition means, out of the area patterns stored in the storage means, and storing means that stores the area pattern of the fingertip coordinate position in association with the hand shape The hand shape stored with the hand shape is estimated as the hand shape in the image, and the estimation result is determined by the hand shape. An image processing method of an image processing apparatus including an estimation unit that outputs as input information, wherein the fingertip coordinate acquisition step acquires a coordinate position of a fingertip of a hand included in the image in the fingertip coordinate acquisition unit (for example, When the image is divided into a plurality of areas in the step S72, S75) of FIG. 8 and the area pattern recognition means, the pattern of the area to which the fingertip coordinates acquired by the processing of the fingertip coordinate acquisition step belongs is used as an area pattern. The area pattern recognition step (for example, step S77 in FIG. 8) to be recognized matches the area pattern recognized by the processing of the area pattern recognition step among the area patterns stored in the storage means in the estimation means. The hand shape stored in association with the area pattern to be Estimated as being the hand shape of the hand at the inner, and a estimation step of outputting the estimation result as the input information by the hand shape (e.g., step S79 of FIG. 8).

[Configuration Example of One Embodiment of Game Table Using Image Processing Device According to the Present Invention]
FIG. 1 is a diagram showing a configuration example of an embodiment of a game machine using an image processing apparatus according to the present invention. The left part in FIG. 1 is a front view of the game table, and the right part is an external perspective view of the game table.

  The game table 11 launches a game ball in accordance with the player's operation, and when the game ball wins a predetermined winning opening, the game ball 11 pays out the corresponding number of game balls, such as a big hit or a small hit. In addition, the game table 11 has a special moving image on the display unit 22 provided in the center of the board surface of the game table 11 in order to enhance the effect when the game ball wins a predetermined winning opening and wins a big hit or a small hit. An image is displayed and a predetermined sound is output. At this time, the game table 11 urges the player to input any one of an operation of changing the hand shape, a tracking operation of the fingertip, and a rotation operation of the fingertip by using a moving image or sound. . In response to such a prompt, when the player performs an input operation prompted by the effect image or sound displayed on the display unit 22 to the camera module 21, the gaming table 11 uses the camera module 21. An image of a hand used for the input operation is captured, the input operation is received by performing appropriate processing on the image, and the effect image and sound are changed according to the received input operation. By accepting such an input operation and reflecting it in the production operation, the player feels as if he / she is manipulating the operation of the game table 11 himself and has a sense of realism in the production effect expressed by the game table. It is possible to have it.

[Example of configuration of game machine]
Next, with reference to FIG. 2, the function implement | achieved by the game stand 11 is demonstrated.

  The gaming table 11 includes a camera module 21 made of CMOS (Complementary Metal Oxide Semiconductor), a display unit 22 made of LCD (Liquid Crystal Display), and the like, and a control unit 31 that controls the overall effects of the gaming table 11. . Although not shown, there are devices that control the operation related to the game, but they are omitted for convenience of explanation.

  As shown in the side sectional view shown in the upper right part of FIG. 3, the camera module 21 includes an imaging element 41 made of COMS at the lower part, an LED 42 for emitting infrared light at the upper part, and a control part 43. In FIG. 3, a side sectional view of the camera module 21 is shown in the upper right part, and a sectional perspective view is shown in the lower left part. As described above, the imaging element 41 of the camera module 21 captures an image made up of infrared light reflected from the hand H by the infrared light generated by the LED 42 being applied to the hand H of the player who is the subject. Then, the captured image is supplied to the control unit 43. As a result, it is possible to capture only the hand image existing in front of the camera module 21. And the control part 43 recognizes the input operation using the hand by the player based on the image supplied from the image sensor 41 and the information on the effect state supplied from the control part 31 that controls the effect. And the information of the input operation which is a recognition result is supplied to the control part 31 which controls an effect.

[Configuration example of camera module]
Next, functions realized by the camera module 21 will be described with reference to FIG. As described with reference to FIG. 3, the camera module 21 includes the image pickup device 41, the LED 42, and the control unit 43. Here, the functions realized by the control unit 43 will be described in detail. It shall be.

  The control unit 43 includes an effect state determination unit 61, an operation condition switching unit 62, a fingertip position calculation unit 63, a fingertip tracking unit 64, a hand shape estimation unit 65, and a rotation operation processing unit 66. The effect state determination unit 61 determines the effect state based on the information on the effect state supplied from the control unit 31 that controls the entire operation of the effect of the game table 11. The operation condition switching unit 62 switches operation conditions accepted as input information based on the determination result of the effect state determination unit 61. Acceptable input information includes input information based on hand shape, input information based on fingertip coordinates, and input information based on rotation direction and rotation speed based on fingertip rotation. Switch which one to accept.

  The fingertip position calculation unit 63 calculates the position of the fingertip coordinates from the image based on the infrared light supplied from the image sensor 41. More specifically, the fingertip position calculation unit 63 includes a binarization processing unit 81, a centroid calculation unit 82, a fingertip candidate area search unit 83, a fingertip position specifying unit 84, a distance calculation unit 85, an LED control unit 86, and an image. An acquisition unit 87 is provided. The LED control unit 86 controls the light emission of the LED 42. The image acquisition unit 87 acquires an image picked up by the image pickup device 41 by being reflected from a hand that is a subject when the LED 42 emits light. The binarization processing unit 81 processes the image acquired by the image acquisition unit 87 into a binary image in which the area of the hand that is the subject is white and the other area is black.

  The center-of-gravity calculation unit 82 calculates the center-of-gravity position of the white region that is the subject in the binarized image. The fingertip candidate area search unit 83 uses the ratio of the white area and the black area within the rectangular range centered on the pixel that is the boundary of the binarized image as the boundary where the black area is larger than the white area. The pixel is searched as a fingertip candidate area. Further, the fingertip candidate area search unit 83 divides the fingertip candidate areas into groups for each area existing in the vicinity. The fingertip position specifying unit 84 controls the distance calculation unit 85 to select the pixel having the longest distance from the barycentric position among the pixels of the fingertip candidate area that constitutes a group for each group of fingertip candidate areas, that is, the barycentric position. And the distance from the center of gravity of the fingertip position specified for each group and the distance from the center of gravity position are output as information on the fingertip position.

  The fingertip tracking unit 64 regards the fingertip position information determined by the fingertip position calculation unit 63 for each group as the fingertip position of the index finger, which is the farthest from the center of gravity position, and sequentially stores the coordinates thereof. Tracked and output as input information. More specifically, the fingertip tracking unit 64 includes a storage unit 121 and a wobble determination unit 122. The fingertip tracking unit 64 stores the coordinates of the immediately previous fingertip position, and then, at a coordinate position within a predetermined distance to from the stored fingertip coordinates and farther than the predetermined distance ti (<to), When the position is detected, the coordinates of the fingertip position to be tracked are assumed to be stored in the storage unit 121 as new coordinates. In addition, the fingertip tracking unit 64 controls the wobble determination unit 122 to assume that the fingertip remains in the same position when it is within the predetermined distance ti from the stored coordinates of the fingertip. It is assumed that the previous coordinate position is stored in the storage unit 121 as it is.

  The hand shape estimation unit 65 estimates the hand shape from the arrangement of the center of gravity position of the hand and the coordinates of the fingertip position based on the information on the fingertip position specified for each group obtained by the fingertip position calculation unit 63, and the hand shape Is output as input information. More specifically, the hand shape estimation unit 65 includes a fingertip position area recognition unit 101, an area pattern storage unit 102, and an estimation unit 103. When the fingertip position area recognizing unit 101 sets an area by dividing an image radially around the center of gravity position based on the information on the fingertip position and the information on the center of gravity output by the fingertip position calculating unit 63, The area pattern including the fingertip position is recognized as an area pattern. The area pattern storage unit 102 stores an area pattern that is an area of the center of gravity position and the fingertip position in advance in association with the hand shape. This area pattern is assumed to be an area pattern of a fingertip position based on a general hand shape, that is, a barycentric position corresponding to a hand shape that can be naturally taken by a hand. The estimation unit 103 compares all the area patterns stored in the area pattern storage unit 102 with the area patterns recognized by the fingertip position area recognition unit 101. The hand shape is the estimation result.

  Based on the information on the fingertip position specified for each group obtained by the fingertip position calculation unit 63, the rotation operation processing unit 66 outputs information on the rotation direction and the rotation speed of the index finger as input information. More specifically, the rotation operation processing unit 66 includes a virtual dial rotation angle measurement unit 141, a non-detection area setting unit 142, a detection area specification unit 143, a detection area storage unit 144, a rotation direction specification unit 145, and a rotation speed calculation unit. 146 and a rotation axis correction unit 147. The non-detection area setting unit 142 sets areas by radially dividing a captured image with a predetermined center position as a reference, and further sets a rectangular range near the center position as a non-detection area. That is, the non-detection area is an area that is regarded as non-detection even if the information on the fingertip position is detected in the area. In addition, when a correction amount for correcting the center position is calculated by an axis correction unit 164 described later, the center positions of the detection area and the non-detection area set immediately before are corrected based on the calculated correction amount. Set the detection area and non-detection area. Based on the information on the fingertip position, the detection area specifying unit 143 specifies which area is detected in a radial pattern with respect to the predetermined center position in the image as described above.

  The detection area storage unit 144 stores the fingertip position information specified by the detection area specifying unit 143 and the area information. The virtual dial rotation angle measurement unit 141 regards each area radially divided with reference to a predetermined position in the image as a virtual dial, and the area immediately before stored in the detection area storage unit 144 When the area of the latest detection position specified by the detection area specifying unit 144 changes, it is considered that the virtual dial has rotated from the difference between the changed areas, and the rotation angle is determined from the difference in angle between the areas. measure. The rotation direction identification unit 145 determines whether the rotation direction is clockwise (clockwise) or counterclockwise (counterclockwise) based on the positional relationship between the immediately preceding detection area and the latest detection area. Identify. The rotation speed calculation unit 146 calculates the rotation speed (angular speed or the number of rotations per unit time) from the change between the immediately preceding detection area and the latest detection area and the time related to the change.

  The rotation axis correction unit 147 corrects the axis that is the center position from the immediately preceding locus based on the information on the fingertip position. More specifically, the rotation axis correction unit 147 includes a timer 161, a rotation determination unit 162, a center of gravity calculation unit 163, and an axis correction unit 164. The timer 161 measures the elapsed time from the timing at which the fingertip position information is first detected. Based on the detection area information stored in the detection area storage unit 144, the rotation determination unit 162 determines whether or not the detected area information is stored for one round, that is, the detection area information is one round. It is determined whether or not there are minutes. The center-of-gravity calculation unit 163, when the rotation determination unit 162 considers that the area information is stored for one round, stores the coordinate information of the fingertip position for one round stored in the detection area storage unit 144. To calculate the position of the center of gravity. The axis correction unit 164 calculates a correction amount for correcting the center position for setting an area at the center of gravity calculated by the center of gravity calculation unit 163.

[Production image display processing]
Next, the effect image display process will be described with reference to the flowchart of FIG.

  In step S11, the production state determination unit 61 performs the production state or the input based on the information indicating the production state supplied from the production control unit 31 or the operation state of the input operation switching operation unit (not shown). Recognize the status of the switching operation.

  In step S12, the effect state determination unit 61 determines whether the recognized effect state or the input switching operation is to receive input information by the fingertip tracking process. In step S12, for example, when the input information by the fingertip tracking process is received, the process proceeds to step S13.

  In step S <b> 13, the operation condition switching unit 62 switches to an operation condition for accepting input information by the fingertip tracking process recognized by the effect state determination unit 61 and accepts input information by the fingertip tracking process.

[Fingertip tracking process]
Here, the fingertip tracking process will be described with reference to the flowchart of FIG.

  In step S41, the production state determination unit 61 notifies the production control unit 31 that the locus when the fingertip is moved toward the image sensor 41 is the input information. In response to this, the control unit 31 controls the display unit 22 to display that the locus when the fingertip is moved toward the image sensor 41 becomes the input information. Thereby, the player recognizes that the trajectory accompanying the movement of the fingertip becomes the input information at any timing.

  In step S42, the LED control unit 86 causes the LED 42 to emit light, and irradiates the vicinity of the position where the player's hand as the subject inputs input information with infrared light. The imaging element 41 captures an image of a player's hand, which is a subject, which is irradiated with infrared light and reflects the irradiated infrared light, for example, an image as shown by an image P1 in FIG. This is supplied to the fingertip position calculation unit 63. An image P1 in FIG. 7 is an image of a hand in which the index finger is raised with the left hand that reflects irradiated infrared light, and a white portion is a hand region, that is, a region that reflects infrared light. The black part is the other area. The image is intended for a moving image. Therefore, the image captured here is an image in units of one frame or one field constituting the moving image.

  In step S <b> 43, the binarization processing unit 81 binarizes the image captured by the image sensor 41. That is, for example, when the image picked up by the image pickup device 41 is the image P1 in FIG. 7, a binary value is obtained by setting a pixel value equal to or greater than a predetermined pixel value as a white pixel value and setting the other as black. The binarization processing unit 81 generates a binarized image as shown by the image P2 in FIG.

  In step S44, the centroid calculating unit 82 calculates the coordinates of the centroid position of the white area indicating the hand area in the binarized image by calculating the following equation (1).

(Xg, Yg) = (1 / n (X ₀ + ... + X _(n-1) ), 1 / n (Y ₀ + ... + Y _(n-1) ))
... (1)

  Here, (Xg, Yg) represents the coordinates of the center of gravity position, and Xi (i = 0, 1, 2,... (N-1)) and Yi (i = 0, 1, 2,. (N-1)) represents the X and Y coordinates of the pixels in the white area.

  That is, for example, in the case of a binary image as shown by the image P2 in FIG. 7, the centroid calculator 82 determines the centroid position (Xg, Yg) of the white area indicating the hand region as shown by the image P3 in FIG. ). Note that the image P3 in FIG. 7 shows an example in which the barycentric coordinates are obtained with the lower left corner of the image as the origin (0, 0), the horizontal direction as the X direction, and the vertical direction as the Y direction. .

  In step S45, the fingertip candidate area search unit 83 centers each pixel for each pixel existing on the boundary between the white area that is the hand area of the binarized image and the black area that is the other area. A rectangular area is set, the ratio between the white area and the black area within the rectangular area is obtained, and the fingertip candidate area is determined by determining whether the pixel is in the vicinity of the fingertip where the ratio of the black area is high. Search for. That is, in the binarized image in which the hand region is a white region and the other region is a black region, as shown in the image P4 in FIG. Since the ratio of black and white in the rectangular area FL1 centered on the pixel is higher than that in the rectangular area FL2 that is not the fingertip candidate area, such a pixel is located in the fingertip portion of the image P5. As indicated by a black dot, the pixel is searched for as a pixel of the fingertip candidate area. Further, the rectangular range FL3 such as noise is removed by this processing.

  In step S46, the fingertip candidate area search unit 83 obtains the distance from the center of gravity for the pixels constituting the searched fingertip candidate area by calculating the following equation (2), and the fingertip candidate area search unit 83 determines the fingertip candidate area according to the distance. Divide into multiple groups of pixels.

Ah = √ ((Xkn−Xg) ² + (Ykn−Yg) ² )
... (2)

  Here, Ah represents the distance from the barycentric coordinates, and (Xkn, Ykn) represents the coordinates of the pixels of the fingertip candidate area. That is, by dividing the pixel coordinates of the fingertip candidate area into groups according to the distance in this way, for example, the coordinates of the pixel of the fingertip candidate area indicated by the black dot in the image P5 in FIG. ), The pixel groups M1 and M2 are classified into groups each having a short distance.

  In step S47, the fingertip candidate area search unit 83 determines, for each pixel group classified according to the distance from the center of gravity, the pixel farthest from the center of gravity position as the representative coordinates of the pixel group as shown in the image P6 of FIG. Set as. That is, as shown in the image P6, the distance between the pixel of each fingertip candidate region and the center of gravity position is obtained, and the pixel farthest from the center of gravity position is set as the representative coordinate of the pixel group.

  In step S <b> 48, the fingertip position specifying unit 84 recognizes the coordinate farthest from the center of gravity among the representative coordinates of each pixel group as the fingertip coordinates. That is, in the case of the image P6 in FIG. 7, the coordinates (Xans, Yans) of the pixel corresponding to the tip of the index finger in the hand image are recognized as the fingertip coordinates. That is, when generating input information based on the trajectory of the fingertip, since the index finger or the like is usually set up and the input information is often constituted by the tip of the index finger, the coordinates of the tip of the index finger are obtained by such processing. It is definitely acquired as input information. However, since the same processing can be performed even if other fingers are raised, if any finger is raised, the coordinates of the tip can be used as input information. In any case, normally, input information corresponding to the input operation at the fingertip that the player would take can be received as coordinate information of the fingertip that moves continuously.

  In step S49, the fingertip tracking unit 64 controls the wobble determination unit 122 to obtain a distance fpt from the coordinates of the fingertip position in the image of the immediately preceding frame or field, and determines whether it is within a predetermined range. . That is, when the fingertip coordinate N1 indicated by the image P7 in FIG. 7 moves to the coordinate N2 as indicated by the image P8, the stagger determination unit 122 obtains the distance fpt between the coordinates N1 and N2. Then, the wobble determination unit 122 changes the coordinates of the fingertip being tracked when the distance fpt is within a predetermined range, that is, longer than the predetermined distance ti and shorter than the predetermined distance to (> ti). And the process proceeds to step S50. That is, when the distance fpt is smaller than the predetermined distance ti, it is regarded that the distance fpt is merely staggered, and when the distance fpt is larger than the predetermined distance to, it is not regarded as continuously moving.

  In step S49, when the distance fpt is longer than the predetermined distance ti and shorter than the predetermined distance to, the fingertip tracking unit 64 determines that the latest fingertip position is continuously moving from the immediately preceding fingertip position. The processing proceeds to step S50. In step S50, the fingertip tracking unit 64 outputs the newly detected coordinates of the fingertip candidate as the fingertip coordinates to be newly tracked.

  In step S51, the fingertip tracking unit 64 stores the detected fingertip coordinates in the storage unit 121 together with information for identifying the current frame or field, and ends the process.

  On the other hand, in step S49, if the distance fpt is not within the predetermined range, that is, shorter than the predetermined distance ti, the player does not intend to move the fingertip and is regarded as wobbling, or If it is longer than the predetermined distance to and cannot be regarded as the movement destination in view of the coordinates of the immediately preceding fingertip, the process proceeds to step S52.

  In step S52, the fingertip tracking unit 64 assumes that there is no change in the coordinate position of the fingertip, that is, the fingertip has not moved, and uses the position coordinate registered in association with the immediately preceding frame or field as the fingertip. Output as the current position coordinates of.

  When moving one of the fingers in the image captured as a subject by the above processing, such as the index finger, the fingertip of the fingertip is continuously captured in a frame unit or a field unit continuously captured by a moving image. The coordinate positions are sequentially tracked, and the coordinates resulting from the tracking are sequentially output as input information. As a result, when the player performs an operation such as drawing a shape such as a circle mark, a triangle mark, or a cross mark with the fingertip in front of the camera module 21, the shape of the trajectory is determined from the tracking result of the coordinates of the fingertip. It is possible to recognize a shape such as a circle, a triangle, or a cross. In addition, it is also possible to recognize the starting point and the ending point from the tracking result of the fingertip coordinates, and to input directions such as up, down, left, and right.

  Now, the description returns to the flowchart of FIG.

  In step S13, fingertip tracking processing is performed, and the coordinates of the fingertips that are input by the player moving the fingertips of the hand sequentially in units of frames or fields are sequentially output as input information.

  In step S <b> 14, the effect state determination unit 61 acquires information on the coordinates of the generated fingertip as input information and supplies it to the effect control unit 31.

  In step S <b> 15, the control unit 31 changes the effect state according to the supplied input information, and displays information according to the changed effect state on the display unit 22.

  On the other hand, when the fingertip coordinates are not used as input information in the fingertip tracking process in step S12, the process proceeds to step S17.

  In step S <b> 17, the effect state determination unit 61 determines whether or not the recognized effect state or the hand shape estimated by the input switching operation by the hand shape estimation process is accepted as input information. In step S17, for example, when the hand shape estimated by the hand shape estimation process is used as input information, the process proceeds to step S18.

  In step S <b> 18, the operation condition switching unit 62 switches to an operation condition for accepting input information by the fingertip tracking process recognized by the effect state determination unit 61 and accepts the hand shape estimated by the hand shape estimation process as input information. .

[Hand shape estimation process]
Here, the hand shape estimation process will be described with reference to the flowchart of FIG.

  In step S71, the effect state determination unit 61 notifies the effect control unit 31 that it is input information to cause the image sensor 41 to image the hand shape. In response to this, the control unit 31 controls the display unit 22 to display that the hand shape to be imaged toward the image sensor 41 is input information, and to indicate the intention to start input by the hand shape. An image that prompts the user to take an image is displayed. Thus, the player can recognize that the input information is configured by an input operation that changes the hand shape at present. The hand shape indicating the input intention is, for example, a predetermined hand shape such as a so-called “goo” hand shape in Janken, and with the predetermined hand shape, the input information in the hand shape is subsequently processed. It is for indicating the start of input.

  In step S <b> 72, the fingertip position calculation unit 63 executes a fingertip coordinate acquisition process to acquire the coordinates that are the fingertip position. This process is the same as the process of steps S42 to S47 described with reference to FIG. That is, the representative coordinates of the coordinate group that is the fingertip candidate area that is the coordinates of the fingertip position are acquired as the coordinates of each fingertip.

  In step S <b> 73, the hand shape estimating unit 65 controls the fingertip position area recognizing unit 101 to determine whether or not the coordinates of the pixel that becomes the fingertip candidate region are not detected. In step S73, for example, when the hand shape is a so-called “goo” shape of Janken, the fingertip position is not detected, so it is considered that the coordinates of the fingertip candidate area are not detected, and the hand shape is Therefore, the process proceeds to step S74.

  In step S <b> 74, the hand shape estimation unit 65 supplies information indicating that the hand shape indicating the input intention in the hand shape is shown to the effect control unit 31. Based on this information, the production control unit 31 generates an image prompting the user to input the hand shape together with information indicating that the input intention in the hand shape has been confirmed, and displays the image on the display unit 22. Let With such a display, the player can recognize the timing at which presentation of a hand shape that is originally desired to be input as input information should be started.

  In step S75, as in the case of step S72, the representative coordinates of the coordinate group serving as the fingertip candidate area serving as the coordinates of the fingertip position are acquired as the coordinates of each fingertip.

  In step S76, the hand shape estimating unit 65 controls the fingertip position area recognizing unit 101 to determine whether or not a coordinate that becomes a fingertip candidate region has been detected. For example, as shown by the image P11 in FIG. 9, when the state where the index finger and the middle finger are raised is presented as a hand shape as so-called “choki” or “V sign”, the coordinates of the fingertip position are: It becomes the tip of the index finger and middle finger. In such a case, it is assumed that the fingertip candidate area has been detected, and the process proceeds to step S77.

  In step S77, the hand shape estimating unit 65 controls the fingertip position area recognizing unit 101, and in which area of the area preset in the input image the representative coordinates of each group of fingertip candidate areas appear. Recognize That is, the area is obtained by dividing the image radially with the center of gravity of the hand-shaped image as the center as shown by the area pattern A1 in FIG. 9, for example, and is divided into eight in FIG. The areas Z0 to Z7 each constitute an area. Further, as shown in the image P11 of FIG. 9, when the index finger and the middle finger are standing upward, the areas Z0 and Z7 are the areas to which the fingertip positions belong as shown in gray in the area pattern A1 of FIG. Is recognized. Note that the number of image divisions for setting the area may be a number other than eight.

  Further, for example, as shown in the image P12 of FIG. 9, when the index finger and the thumb are set up and the index finger points to the left in the figure, the fingertip position is indicated as shown in gray in the area pattern A2. Areas Z5 and Z7 are recognized as the area to which they belong. Similarly, for example, as shown in the image P13 of FIG. 9, when the index finger and the middle finger are raised and both the index finger and the middle finger are pointing leftward in the figure, the area pattern A3 is shown in gray. In addition, areas Z5 and Z6 are recognized as areas to which the fingertip position belongs.

  That is, as shown in images P11 and P13 in FIG. 9, even if the hand shape is the same, other areas can be obtained by changing the direction and direction of the hand shape by dividing the hand shape into eight as described above. It is possible to configure a pattern, and it is possible to express various area patterns by simply changing the direction even by simply configuring a hand shape. In addition, what should be expressed is input information by hand shape, but since it is essentially a combination of fingertip area patterns, various types of information can be used as long as the area pattern can be expressed without creating a complicated hand shape. Simple patterns can be expressed.

  In step S <b> 78, the estimation unit 103 registers one that matches the area pattern recognized by the fingertip position area recognition unit 101 among the area patterns previously registered for each input information in the area pattern storage unit 102. It is determined whether or not. If there is a matching area pattern in step S78, in step S79, the estimation unit 103 outputs the input information registered in association with the matched area pattern to the control unit 31 as an estimation result.

  In step S80, the control unit 31 displays the input information supplied from the estimation unit 103 as an estimation result on the display unit 22.

  On the other hand, if the fingertip candidate area is not detected in step S76, in step S81, the hand shape estimation unit 65 determines whether or not a predetermined time has elapsed since the input intention in the hand shape has been confirmed. If the time has not elapsed, the process returns to step S74. That is, until it is determined that there is no input of the hand shape until the predetermined time has elapsed, the processes in steps S74 to S76, S81 are repeated, and the acceptance of the input in the hand shape is continued.

  If it is determined in step S81 that the predetermined time has elapsed, or if no fingertip candidate area exists in step S73, the process proceeds to step S82.

  In step S82, it is determined whether or not a predetermined time has elapsed since the user's intention to input with a hand shape is prompted. If it is determined in step S82 that the predetermined time has not elapsed since the input intention with the hand shape is prompted, the process returns to step S71. In other words, an image prompting the user to input his / her hand shape is displayed again, and the processing from the beginning is performed. On the other hand, in step S82, when a predetermined time elapses after prompting the user to indicate the input intention in the hand shape, or in step S78, the area pattern registered in advance for each input information in the area pattern storage unit 102. If there is no registered pattern that matches the area pattern recognized by the fingertip position area recognition unit 101, the process proceeds to step S83.

  In step S83, the estimation unit 103 outputs the input information of the area pattern registered in association with a predetermined hand shape to the control unit 31 as an estimation result.

  In step S84, the control unit 31 displays the input information of the area pattern registered in association with the predetermined hand shape supplied from the estimation unit 103 as the estimation result on the display unit 22.

  In other words, even if input by hand shape is not seen, the processing proceeds by assuming that there is input by a predetermined hand shape. This is because, in general, in the production of a game stand, the next production operation cannot be realized unless there is some input information, and the player performs an input operation with a hand shape for some reason. This is to prevent the production operation from stopping even if it is not.

  With the above processing, it is possible to input various input information simply by creating a simple hand shape with one hand and directing it in an appropriate direction to form an area pattern. That is, in the above-described embodiment, when the hand-shaped center of gravity is divided into 8 areas radially, 56 ways (= 8! / (3! ×) are obtained by using 5 fingers with one hand. 5!)), And it becomes possible to input a large amount of input information only with a simple hand-shaped configuration. Furthermore, it is possible to input more input information with an input operation with a simple non-contacting hand shape with one hand by composing input information by continuously combining a plurality of hand shape estimation results. .

  Now, the description returns to the flowchart of FIG.

  When the input information corresponding to the hand shape estimated by the hand shape estimation process in step S18 is output, in step S19, the control unit 31 acquires input information corresponding to the estimated hand shape.

  On the other hand, if it is determined in step S17 that the input information is not received by the hand shape estimation process, the process proceeds to step S20.

  In step S20, the effect state determination unit 61 determines whether or not the recognized effect state or the input switching operation is to receive input information by the rotation detection process. In step S20, for example, when the input information by the rotation detection process is received, the process proceeds to step S21.

  In step S <b> 21, the operation condition switching unit 62 switches to an operation condition for accepting input information based on the rotation detection process recognized by the effect state determination unit 61 and accepts input information based on the rotation detection process.

[Rotation detection processing]
Here, the rotation detection process will be described with reference to the flowchart of FIG.

  In step S <b> 101, the effect state determination unit 61 supplies the control unit 31 with accepting input information based on the rotation operation. Based on this information, the control unit 31 is in a state of accepting input information based on the rotation operation. The control unit 31 generates an image for prompting the input operation, such as rotating the fingertip and causing the image sensor 41 to capture an image. To display.

  In step S102, the rotation operation processing unit 66 controls the virtual dial rotation angle measurement unit 141 to initialize the rotation angle θ of the virtual dial to 0 degrees. Here, the virtual dial is divided into eight areas radially from a predetermined center position in the image shown by the area pattern A21 in FIG. 11, and each non-detection area is set at the center portion. The area is like a dial. The setting of the areas Z0 to Z7 in the area pattern A21 is the same as that described with reference to FIG. 9 except for the non-detection area.

  In step S103, the non-detection area setting unit 142 sets and initializes the detection area and the non-detection area. In addition, based on the center position set by the area correction process to be described later, the areas are radially divided as indicated by an area pattern A21 in FIG. Here, each area is set by dividing it into 8 sections, 45 degrees with respect to the center. Further, as indicated by an area pattern A21 in FIG. 11, the non-detection area is set as a rectangular area (gray portion) near the center. In the non-detection area, the coordinates of the fingertip are detected in the same manner as in the case of detecting the hand shape in the case of FIG. 9 described above, but here it is set as an area to be processed by assuming that it is not detected. The Further, when the correction amount is obtained by an area correction process described later, the non-detection area setting unit 142 shifts the detection area set immediately before and the center position of the non-detection area by the correction amount to Set the detection area and detection area.

  In step S104, the fingertip position calculation unit 63 executes a fingertip tracking process to detect the longest fingertip coordinate among the hand shapes, that is, the farthest coordinate from the gravity center position of the hand shape. Note that the fingertip tracking process is the same as the process described with reference to the flowchart of FIG.

  In step S105, the fingertip position calculation unit 63 determines whether or not the fingertip coordinates are detected. If the fingertip coordinates are not detected, the process returns to step S104. That is, the processes in steps S104 and S105 are repeated until the coordinates of the fingertip are detected. If the fingertip coordinates are detected in step S105, the process proceeds to step S106.

  In step S106, the detection area specifying unit 143 specifies the coordinates of the fingertips detected by the fingertip position calculating unit 63 and the detection area Ar1, and stores them in the detection area storage unit 144 together with the detected time t1.

  In step S107, the fingertip position calculation unit 63 executes a fingertip tracking process to detect the coordinate of the longest fingertip among the hand shapes. Note that the fingertip tracking process is the same as the process described with reference to the flowchart of FIG.

  In step S108, the fingertip position calculation unit 63 determines whether or not the fingertip coordinates are detected. If the fingertip coordinates are not detected, the process returns to step S107. That is, the processes of steps S107 and S108 are repeated until the fingertip coordinates are detected. If the fingertip coordinates are detected in step S108, the process proceeds to step S109.

  In step S109, the detection area specifying unit 143 specifies the coordinates of the fingertips detected by the fingertip position calculation unit 63 and the detection area Ar2, and stores them in the detection area storage unit 144 together with the detected time t2.

  In step S110, the rotation direction specifying unit 145 determines whether or not the detected fingertip coordinate areas at the times t1 and t2 stored in the detection area storage unit 144 match. That is, for example, in the case of the area pattern A21 in FIG. 11, when the coordinate position r2 of the fingertip is detected at time t1, and the coordinate position r3 of the fingertip is detected at time t2, the detected area is the area. Since it becomes Z1, it is considered that the detection areas coincide with each other, and the process proceeds to step S116.

  On the other hand, in the case of the area pattern A21 of FIG. 11, when the coordinate position r5 of the fingertip is detected at time t1, and the coordinate position r6 of the fingertip is detected at time t2, the detected areas Ar1 and Ar2 Since it becomes Z2 and Z3, it is regarded that the detection areas do not coincide with each other, and the process proceeds to step S111.

  In step S111, the rotation direction specifying unit 145 determines whether or not the detection area Ar2 is an adjacent area in the right rotation direction of the detection area Ar1. That is, in the case of the area pattern A21 in FIG. 11, when the coordinate position r5 of the fingertip is detected at time t1, and the coordinate position r6 of the fingertip is detected at time t2, the detected areas Ar1 and Ar2 Since it becomes Z2 and Z3, the detection area Ar2 = Z3 is an adjacent area in the right rotation direction of the detection area Ar1 = Z2, and therefore the process proceeds to step S112.

  In step S112, the rotation direction specifying unit 145 considers the rotation direction of the fingertip to be clockwise (clockwise).

  In step S113, the rotation speed calculation unit 146 calculates the angular velocity of the rotation angle. That is, in the case of FIG. 11, since 45 degrees are divided radially from the center position, the change in units of 45 degrees is calculated as the angular velocity formed by the change time (t2-t1). Note that the information is not limited to the angular speed as long as it can be recognized as the rotational speed. For example, the rotational speed per unit time may be obtained instead of the angular speed.

  In step S114, the virtual dial rotation angle measurement unit 141 increments the virtual dial rotation angle θ by the changed angle. That is, in this case, since the virtual dial is only moved to the adjacent area and the rotation angle is 45 degrees, the virtual dial rotation angle θ is set to (θ + 45) degrees.

  In step S115, the detection area specifying unit 144 updates the detection area Ar1 to the detection area Ar2, and updates the time t1 to the time t2.

  In step S116, it is determined whether or not the input information input process by the rotation detection process has been completed. If the input information has not been completed, the process returns to step S107. If the process has been completed, the process ends.

  In step S111, if the detection area Ar2 is not an adjacent area in the right rotation direction of the area Ar1, that is, if it is not clockwise (clockwise), the process proceeds to step S117.

  In step S117, the rotation direction identification unit 145 determines whether or not the detection area Ar2 is an adjacent area in the left rotation direction of the detection area Ar1. That is, in the case of the area pattern A21 of FIG. 11, when the coordinate position r6 of the fingertip is detected at time t1, and the coordinate position r5 of the fingertip is detected at time t2, the detected areas Ar1 and Ar2 Since Z3 and Z2 are detected, the detection area Ar2 = Z2 is an adjacent area in the left rotation direction of the detection area Ar1 = Z3. Therefore, the process proceeds to step S118.

  In step S118, the rotation direction specifying unit 145 considers the fingertip movement to be counterclockwise (counterclockwise), and the process proceeds to step S113.

  In step S117, if the detection area Ar2 is not an adjacent area in the counterclockwise direction of the detection area Ar1, that is, it is not clockwise or counterclockwise, it is regarded as not rotating, and the process proceeds to step S116. move on.

  While the detection areas are sequentially switched according to the rotation of the fingertip coordinates by the above processing, the rotation direction and the rotation speed are detected when the areas are switched, and the rotation angle is counted. As a result, for example, when the player performs an operation of rotating the fingertip in front of the camera module 21, it is shown in the upper part of FIG. 12 based on the rotation angle, the rotation direction, and the rotation speed detected by the rotation operation of the fingertip. As in the image 22A, it is possible to generate an effect image such that the frames a to h are rotated as indicated by arrows in the drawing and displayed on the display unit 22 based on the rotation angle, rotation direction, and rotation speed. Become. Similarly, for example, according to the rotation angle, the rotation direction, and the rotation speed, a heart mark such as the image 22B shown in the lower part of FIG. It is possible to generate an effect image such that the direction, position, and speed at which the heart marks spread or gather are changed in accordance with the rotation of the fingertip. Furthermore, the effect images are not limited to these, and other effect images may be configured using the rotation angle, the rotation direction, and the rotation speed as parameters.

[Area correction processing]
Next, the area correction process will be described with reference to the flowchart of FIG.

  In step S141, the rotation axis correction unit 147 determines whether or not the rotation detection process is executed by the effect state determination unit 61, and repeats the same process until the rotation detection process is executed. For example, when the process of the flowchart of FIG. 10 described above is executed, the process proceeds to step S142.

  In step S142, the rotation axis correction unit 147 controls the timer 161 to initialize the timer counter Tm.

  In step S143, the rotation axis correction unit 147 increments the counter Tm of the timer 16 by 1.

  In step S144, the coordinate position of the fingertip is detected by the fingertip coordinate tracking process by the fingertip position calculation unit 63. This process is the same as the process described with reference to the flowchart of FIG.

  In step S145, it is determined whether or not the coordinates of the fingertip have been detected. If the detection area is not detected, the process returns to step S143. That is, the processes in steps S143 to S145 are repeated until the fingertip coordinates are detected.

  In step S146, when the coordinates of the fingertip are detected in the detection area, the detection area storage unit 144 stores the detected area.

  In step S147, the detection area storage unit 144 stores the detected fingertip coordinates.

  In step S148, the rotation determination unit 162 refers to the counter Tm of the timer 161 to determine whether or not a predetermined time has elapsed since the counter Tm exceeded a predetermined value and rotation started. In step S148, for example, when the predetermined time has not elapsed, the process proceeds to step S149.

  In step S149, the rotation determination unit 162 rotates the fingertip once based on the information stored in the detection area storage unit 144, and all areas are registered as detection areas. It is determined whether or not is registered. If it is determined in step S149 that the rotation has not been completed, the process returns to step S143. That is, the processes in steps S143 to S149 are repeated until it is considered that the rotation has been completed.

  If it is determined in step S149 that the rotation has been made once, the process proceeds to step S150.

  In step S150, the center-of-gravity calculation unit 163 reads information for one round of the coordinates of the fingertip stored in the detection area storage unit 144, and obtains the position of the center of gravity.

  In step S151, the axis correction unit 164 obtains a deviation between the center position constituting the current area pattern and the obtained center position and corrects the center position constituting the area pattern to move to the center position. Find the correction amount.

  That is, as shown in the upper part of FIG. 11, the center position of the area pattern A21 is different from the center of gravity position O obtained as the center of gravity of the coordinate positions r1 to r13. Therefore, the axis correction unit 164 obtains the deviation between the center position and the center of gravity position O, and the correction amount is such that the position obtained as the center of gravity position is the center as shown by the area pattern A22 in the lower part of FIG. Ask for. When the correction amount is used and the detection area and the non-detection area are set immediately before in the process of step S103 in the flowchart of FIG. 10 described above, the center position is not detected at a position shifted by the correction amount. An area and a detection area are set.

  Based on the coordinates of the detection position detected by the rotation in this way, the difference between the center of the detection area and the non-detection area where the area pattern is set and the center of gravity position of the fingertip coordinates actually detected is obtained as a correction amount. Since the center position is corrected by the correction amount, it is possible to appropriately correct the deviation of the rotation position of the fingertip with individual differences, and more accurately measure the rotation angle, rotation direction, and rotation speed. It becomes possible.

  Now, the description returns to the flowchart of FIG.

  In step S21, rotation detection processing is performed, and a rotation angle, a rotation direction, and a rotation speed that are input when the player rotates the fingertip of the hand are sequentially output as input information.

  In step S <b> 22, the effect state determination unit 61 acquires input information including a rotation direction and a rotation speed obtained by the generated rotation of the fingertip, and supplies the input information to the effect control unit 31.

  Furthermore, in step S20, for example, when the input information by the rotation detection process is not received, the production state determination unit 61 generates specific input information and supplies it to the control unit 31 in step S23. By this process, the control unit 31 configures the effect image as if specific input information was input. In other words, as described above, some effect images stop unless some input information is entered, and thus such processing is executed in order to prevent the effect image from stopping.

  Based on the hand image picked up by the above processing, input information for moving the coordinate position of the fingertip, input information by the hand shape, and input information by the rotation operation by the fingertip can be switched and accepted. As a result, the player can easily input the input information in a non-contact manner only by using the hand.

  In the above description, an example has been described in which the fingertip is imaged by the camera module, the trajectory of the fingertip moving is tracked, the hand shape is estimated, or the rotation is detected. As long as the image can be picked up with a hand, it is not limited to a hand or a fingertip, and other things may be used. For example, instead of a fingertip, a rod-shaped object may be held and used like a fingertip. In this case, since the rod-shaped tip portion farthest from the barycentric position of the rod-shaped object itself is recognized as the fingertip, the same effects as the above-described processing can be achieved. If the camera module can be imaged with infrared light even with a fingertip or hand shape with a glove or magic hand but not with a hand, for example, the same action as described above , Can have an effect.

  By the way, the series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

  FIG. 14 shows a configuration example of a general-purpose personal computer. This personal computer incorporates a CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via the bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

  The input / output interface 1005 includes an input unit 1006 including an input device such as a keyboard and mouse for a player to input an operation command, an output unit 1007 for outputting a processing operation screen and an image of the processing result to a display device, a program, and various data. Are connected to a storage unit 1008 including a hard disk drive and the like, and a local area network (LAN) adapter and the like, and a communication unit 1009 that executes communication processing via a network represented by the Internet. Also, a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor A drive 1010 for reading / writing data from / to a removable medium 1011 such as a memory is connected.

  The CPU 1001 is read from a program stored in the ROM 1002 or a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage unit 1008, and loaded from the storage unit 1008 to the RAM 1003. Various processes are executed according to the program. The RAM 1003 also appropriately stores data necessary for the CPU 1001 to execute various processes.

  In this specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series in the order described, but of course, it is not necessarily performed in time series. Or the process performed separately is included.

11 gaming table 21 camera module 22 display unit 31 control unit 41 imaging unit 42 LED
Reference Signs List 43 control unit 61 effect state determination unit 62 operation condition switching unit 63 fingertip position calculation unit 64 fingertip tracking unit 65 hand shape estimation unit 66 rotational motion processing unit

Claims (4)

An image processing apparatus for receiving input information of the player by a hand shape of the player's hand included in the image,
Fingertip coordinate acquisition means for acquiring the coordinate position of the fingertip of the hand included in the image;
An area pattern recognizing unit for recognizing a pattern of an area to which the fingertip coordinates acquired by the fingertip coordinate acquiring unit belong as an area pattern when dividing the image into a plurality of areas;
Storage means for storing an area pattern of fingertip coordinate positions in association with the hand shape;
Of the area patterns stored in the storage unit, the hand shape stored in association with the area pattern that matches the area pattern recognized by the area pattern recognition unit is the hand shape in the image. And estimating means for estimating as one and outputting the estimation result as input information by hand shape ,
When the area pattern recognized by the area pattern recognition means matches the area pattern stored in association with a predetermined hand shape among the hand shapes included in the storage means, the estimation means In the following, the area pattern recognized by the area pattern recognition unit is any of the area patterns stored in association with all the hand shapes included in the storage unit. An image processing apparatus that estimates that the hand shape is stored in association with the matched area pattern and outputs the hand shape that is the estimation result as input information when it matches . The estimation means has a specific hand shape when the area pattern recognized by the area pattern recognition means does not match the area pattern stored in association with all the hand shapes included in the storage means. The image processing apparatus according to claim 1, wherein the hand shape as an estimation result is output as input information. An image processing apparatus for receiving input information of the player by a hand shape of the player's hand included in the image,
Fingertip coordinate acquisition means for acquiring the coordinate position of the fingertip of the hand included in the image;
An area pattern recognizing unit for recognizing a pattern of an area to which the fingertip coordinates acquired by the fingertip coordinate acquiring unit belong as an area pattern when dividing the image into a plurality of areas;
Storage means for storing an area pattern of fingertip coordinate positions in association with the hand shape;
Of the area patterns stored in the storage unit, the hand shape stored in association with the area pattern that matches the area pattern recognized by the area pattern recognition unit is the hand shape in the image. And estimating means for estimating as one and outputting the estimation result as input information by hand shape ,
When the area pattern recognized by the area pattern recognition means matches the area pattern stored in association with a predetermined hand shape among the hand shapes included in the storage means, the estimation means In the following, the area pattern recognized by the area pattern recognition unit is any of the area patterns stored in association with all the hand shapes included in the storage unit. An image processing method of an image processing apparatus for estimating that the hand shape is stored in association with the matched area pattern, and outputting the hand shape as an estimation result as input information ,
A fingertip coordinate obtaining step for obtaining a coordinate position of a fingertip of a hand included in the image in the fingertip coordinate obtaining means;
In the area pattern recognition means, when dividing the image into a plurality of areas, an area pattern recognition step for recognizing an area pattern to which the fingertip coordinates acquired by the processing of the fingertip coordinate acquisition step belong as an area pattern;
The hand shape stored in association with the area pattern that matches the area pattern recognized by the process of the area pattern recognition step among the area patterns stored in the storage means in the estimation means is stored in the image. An image processing method comprising: an estimation step that estimates a hand shape of the hand inside and outputs an estimation result as input information based on the hand shape. An image processing apparatus for receiving input information of the player by a hand shape of the player's hand included in the image,
Fingertip coordinate acquisition means for acquiring the coordinate position of the fingertip of the hand included in the image;
An area pattern recognizing unit for recognizing a pattern of an area to which the fingertip coordinates acquired by the fingertip coordinate acquiring unit belong as an area pattern when dividing the image into a plurality of areas;
Storage means for storing an area pattern of fingertip coordinate positions in association with the hand shape;
Of the area patterns stored in the storage unit, the hand shape stored in association with the area pattern that matches the area pattern recognized by the area pattern recognition unit is the hand shape in the image. And estimating means for estimating as one and outputting the estimation result as input information by hand shape ,
When the area pattern recognized by the area pattern recognition means matches the area pattern stored in association with a predetermined hand shape among the hand shapes included in the storage means, the estimation means In the following, the area pattern recognized by the area pattern recognition unit is any of the area patterns stored in association with all the hand shapes included in the storage unit. A computer that controls the image processing apparatus that outputs the hand shape that is the estimation result as input information, presuming that the hand shape is stored in association with the matched area pattern .
A fingertip coordinate obtaining step for obtaining a coordinate position of a fingertip of a hand included in the image in the fingertip coordinate obtaining means;
In the area pattern recognition means, when dividing the image into a plurality of areas, an area pattern recognition step for recognizing an area pattern to which the fingertip coordinates acquired by the processing of the fingertip coordinate acquisition step belong as an area pattern;
The hand shape stored in association with the area pattern that matches the area pattern recognized by the process of the area pattern recognition step among the area patterns stored in the storage means in the estimation means is stored in the image. A program for executing a process including: an estimation step for estimating a hand shape in the hand and outputting an estimation result as input information based on the hand shape.

Images