JP7209495B2 - Image processing device, image processing method - Google Patents

Description

The present invention relates to technology for presenting mixed reality.

2. Description of the Related Art In recent years, much research has been conducted on a mixed reality (MR) system for the purpose of seamlessly connecting a real space and a virtual space. As an image display device for presentation in these systems, for example, a head mounted display (HMD) can be used. With the progress of MR system research, masking techniques have also been proposed for the purpose of synthesizing a real object and a virtual object, as in Patent Document 1 and Non-Patent Document 1.

Furthermore, in order to solve the problem that the HMD wearer's field of vision is covered and the operation switches cannot be seen, masking technology is used to detect a part of the user's body such as the user's hand, and gesture operation technology that manipulates virtual objects has been developed. Development is also underway. Japanese Patent Laid-Open No. 2002-200003 discloses a technique for detecting a predetermined movement of a part of the user's body corresponding to a display element by a detection unit that detects movement of an object, and activating processing. Japanese Patent Application Laid-Open No. 2002-200001 discloses a technique of stacking virtual panels in the depth direction, detecting the depth direction position and movement of a hand, and selecting a predetermined panel from a plurality of virtual panels.

Japanese Patent Application Laid-Open No. 2002-157606 JP-A-8-6708 Patent No. 5262681

Kenichi Hayashi, Hirokazu Kato, Shogo Nishida, “Front/back determination of real and virtual objects using boundary line-based stereo matching”, Transactions of the Virtual Reality Society of Japan, Vol. 10, No. 3, pp. 371-380, 2005.

In the methods of Patent Documents 1 and 2, hand and finger recognition processing is performed and then hand and finger motions are determined. May be difficult to implement. In addition, since recognition of hands and fingers is assumed, operation becomes difficult under conditions in which shape recognition of hands and fingers is unstable.

The present invention has been made in view of such problems, and provides a technique for detecting a target object in an image more simply and stably and executing processing based on the detected target object. .

One aspect of the present invention is an image processing device that outputs an image to a head-mounted display device having an imaging unit,
a selection means for selecting, as an operation point, a point that satisfies a prescribed condition from among a group of points forming the contour of the target object in the captured image captured by the imaging unit;
and processing means for executing processing based on the operating point ,
The selection means selects, as the operation point, the uppermost point in the visual field of the head-mounted display device from the point group .

According to the configuration of the present invention, it is possible to more easily and stably detect a target object in an image, and execute processing based on the detected target object.

FIG. 2 is a block diagram showing a system configuration example; 4 is a flowchart of processing performed by the information processing apparatus 1000; FIG. 3 is a diagram for explaining processing according to the flowchart of FIG. 2; 4 is a flowchart of processing performed by the information processing apparatus 1000; FIG. 5 is a view for explaining processing according to the flowchart of FIG. 4; 4 is a flowchart of processing performed by the information processing apparatus 1000; FIG. 7 is a diagram for explaining processing according to the flowchart of FIG. 6; 4 is a flowchart of processing performed by the information processing apparatus 1000; FIG. 9 is a diagram for explaining processing according to the flowchart of FIG. 8; 4 is a flowchart of processing performed by the information processing apparatus 1000; FIG. 11 is a diagram for explaining processing according to the flowchart of FIG. 10; 4 is a flowchart of processing performed by the information processing apparatus 1000; FIG. 13 is a diagram for explaining processing according to the flowchart of FIG. 12; 4 is a flowchart of processing performed by the information processing apparatus 1000; FIG. 15 is a diagram for explaining processing according to the flowchart of FIG. 14; FIG. 2 is a block diagram showing a hardware configuration example of a computer device; FIG. 2 is a block diagram showing a system configuration example; 4 is a flowchart of processing performed by the information processing apparatus 1000; FIG. 19 is a diagram for explaining processing according to the flowchart of FIG. 18; 4 is a flowchart of processing performed by the information processing apparatus 1000; FIG. 21 is a diagram for explaining processing according to the flowchart of FIG. 20;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the embodiment described below is an example of a specific implementation of the present invention, and is one of specific examples of the configuration described in the claims.

[First Embodiment]
First, a configuration example of a system according to this embodiment will be described using the block diagram of FIG. As shown in FIG. 1, the system according to the present embodiment includes an information processing device 1000, which is a computer device such as a PC (personal computer) or a tablet terminal device, and an HMD 1100 as an example of a head-mounted display device. . The information processing apparatus 1000 and the HMD 1100 are connected so as to be able to perform data communication with each other, for example, via a network such as a LAN or the Internet. Note that the form of connection between the information processing apparatus 1000 and the HMD 1100 is not limited to a specific form of connection, and may be, for example, wireless or wired.

First, the HMD 1100 will be explained. The image capturing unit 1110 includes a left eye image capturing unit (left viewpoint) for capturing a real space image (left eye image) to be provided to the left eye of the user wearing the HMD 1100 on the head, and and a right-eye imaging unit (right viewpoint) for capturing an image of the physical space to be provided (right-eye image). Each of the left-eye imaging section and the right-eye imaging section may be an imaging section that captures a still image, or may be an imaging section that captures a moving image. Then, the imaging unit 1110 transmits a set of the left-eye image captured by the left-eye imaging unit and the right-eye image captured by the right-eye imaging unit to the information processing apparatus 1000 as a stereo image (that is, a stereo image). The image may be a still image or a moving image).

The display unit 1120 includes a left-eye display unit for providing images and characters to the left eye of the user wearing the HMD 1100 on the head, and a right-eye display unit for providing images and characters to the right eye of the user. and The display unit for the left eye and the display unit for the right eye are attached to the HMD 1100 so as to be positioned in front of the left and right eyes of the user wearing the HMD 1100 on the head, respectively. display the image or text that has been displayed.

Next, the information processing device 1000 will be described. The information processing device 1000 is a device that functions as an image processing device capable of executing each process described later. Acquisition unit 1010 receives the stereo image sent from imaging unit 1110 . The contour generation unit 1020 forms the contour of a target object (a real object such as a human hand that the user uses to perform some operation) shown in the stereo image (in the captured image) received by the acquisition unit 1010. Find the 3D position of each point.

Using the image for the left eye received by the acquisition unit 1010, the measurement unit 1030 sets the world coordinate system (one point in the physical space as the origin, and the three axes orthogonal to each other at the origin as the x-axis, the y-axis, and the y-axis). The position and orientation of the left viewpoint in the coordinate system with the z-axis is obtained. Further, the measurement unit 1030 uses the right-eye image received by the acquisition unit 1010 to obtain the position and orientation of the right viewpoint in the world coordinate system. For example, the measurement unit 1030 captures the image based on feature points (markers intentionally placed by a person in the real space, natural features that originally exist in the real space, etc.) appearing in the image. Calculate the position and orientation of the imaging unit. Note that there are various methods for acquiring the position and orientation of the imaging unit in the world coordinate system, and the method is not limited to a specific method. For example, a sensor whose position and orientation relative to the imaging units (left-eye imaging unit and right-eye imaging unit) are known is attached to the HMD 1100, and the measured value by the sensor is converted based on the relative position and orientation. , the position and orientation of the imaging unit in the world coordinate system may be obtained. Alternatively, the position and orientation of the imaging unit in the world coordinate system may be obtained by a method using a motion capture system, a method using Simultaneous Localization and Mapping (SLAM) using captured images, or the like.

The operating point calculation unit 1040 selects, as operating points, points that satisfy a prescribed condition from among the points that form the contour line. In the present embodiment, the operating point calculation unit 1040 selects, as an operating point, the point located on the uppermost side of the image in the row direction among the points forming the contour line.

The signal generator 1050 generates a trigger signal for executing prescribed processing based on the three-dimensional position and movement of the operating point, the position and movement of the operating point relative to the prescribed virtual object, and the like. and sent to the signal processing unit 1070 . In this embodiment, when the pointer placed at the position of the operation point touches a prescribed portion of the button (virtual object), it is determined that the button has been pressed, and a signal serving as a trigger for executing prescribed processing. is generated and sent to the signal processing unit 1070 . However, the conditions for generating a trigger signal for executing prescribed processing are not limited to specific conditions. For example, when recognizing a change in the time-series 3D position of an operating point (either an absolute position or a 3D position relative to a specified virtual object) as a specified gesture, A trigger signal may be generated and sent to the signal processing unit 1070 . Signal processing section 1070 executes processing corresponding to the signal sent from signal generating section 1050 .

The holding unit 1060 holds drawing data of each virtual object (CG model) placed in the virtual space. The drawing data of the virtual object is data necessary for drawing the virtual object. For example, when a virtual object is composed of polygons, the rendering data of the virtual object includes the color, material, and normal vector of the polygon, the three-dimensional position of each vertex that constitutes the polygon, the texture mapped to the polygon, and the world of the virtual object. It contains data such as position and orientation in the coordinate system. The virtual object placed in the virtual space includes a pointer placed at the three-dimensional position of the operation point to allow the user to visually recognize the three-dimensional position of the operation point, and the button described above.

The CG generation unit 1080 uses the drawing data of each virtual object held in the holding unit 1060 to construct a virtual object and place it in the virtual space. Based on the position and orientation of the left viewpoint in the world coordinate system obtained by the measurement unit 1030, the CG generation unit 1080 generates an image of the virtual space seen from the left viewpoint (virtual space image for the left eye). Furthermore, the CG generation unit 1080 generates a virtual space image (right eye virtual space image) viewed from the right viewpoint based on the position and orientation of the right viewpoint in the world coordinate system obtained by the measurement unit 1030 . Techniques for generating an image of a virtual space viewed from a viewpoint having a prescribed position and orientation are well known, and therefore descriptions of these techniques will be omitted.

The synthesizing unit 1090 generates a synthetic image obtained by superimposing the virtual space image for the left eye on the image for the left eye received by the acquiring unit 1010 as the mixed reality space image for the left eye. Further, the synthesizing unit 1090 generates a synthesized image by superimposing the virtual space image for the right eye on the image for the right eye received by the acquiring unit 1010 as the mixed reality space image for the right eye. At this time, when the virtual space image is superimposed on part or all of the image area in which the target object is shown in each of the image for the left eye and the image for the right eye, the depth between the target object and the virtual object should be considered. to match the obscurance relationships between objects. In other words, superimposition of a virtual space image is permitted for a partial area in which a virtual object closer to the viewpoint than the target object overlaps in the image area of the target object. On the other hand, superimposition of a virtual space image is prohibited for a partial area covered by a virtual object farther from the viewpoint than the target object in the image area of the target object.

The synthesizing unit 1090 sends the left-eye mixed reality space image to the left-eye display unit of the display unit 1120 of the HMD 1100 , and sends the right-eye mixed reality space image to the display unit 1120 of the HMD 1100 . is sent to the display unit for the right eye.

Next, the information processing apparatus 1000 generates a mixed reality space image for one frame (a mixed reality space image for the left eye and a mixed reality space image for the right eye) and outputs the mixed reality space image to the display unit 1120 . Processing will be described according to the flowchart of FIG. By repeatedly performing the processing according to the flowchart of FIG. 2, it is possible to generate multiple frames of mixed reality space images and send them to the HMD 1100 .

In step S<b>2000 , acquisition section 1010 receives the stereo image sent from imaging section 1110 . In step S2100, contour generation section 1020 extracts the contour lines (two-dimensional contour lines) of the target object on each of the left-eye image and the right-eye image included in the stereo image. For example, as shown in FIG. 3A, when extracting a two-dimensional contour line of a hand 3010 from an image 3000 including a human hand 3010 as a target object, the image area of the hand is extracted from the captured image of the hand in advance. is extracted and registered in the table. The color information may be RGB, or may be represented by YCbCr luminance and tint information. The method for acquiring the two-dimensional outline is not limited to this, and a segmentation method using graph cutting, machine learning, or the like may be used. Then, as shown in FIG. 3B, the outline 3020 of the image area having the color information registered in the table is extracted as the two-dimensional outline of the hand 3010 from the image 3000 .

In step S2200, contour generating section 1020 generates a 3D contour of the target object by matching the 2D contour extracted from the image for left eye and the 2D contour extracted from the image for right eye. do. The method of generating the three-dimensional outline is not limited to a specific method, and for example, the method disclosed in Non-Patent Document 1 can be applied. That is, the epipolar line corresponding to the sampling points on the two-dimensional contour line on one of the images for the left eye and the image for the right eye included in the stereo image is projected onto the other image. A point at which the dimensional contour line intersects is defined as a corresponding point. A plurality of sampling points on the two-dimensional contour are determined to obtain a plurality of corresponding points on the two-dimensional contour. Then, the depth value of each of the plurality of corresponding points on the two-dimensional contour is calculated by triangulation.

Next, in step S2300, operating point calculation section 1040 selects, as an operating point, the corresponding point having the uppermost coordinate value in the row direction in the image from which the corresponding points have been acquired. For example, as shown in FIG. 3C, among the corresponding points (indicated by "○") on the three-dimensional contour 3030 corresponding to the contour 3020 in FIG. A corresponding point 3040 with a value is selected as the operating point. Note that if there are a plurality of real space objects from which the operating points are obtained, one real space object may be selected for extracting the operating points.

In this embodiment, among a plurality of three-dimensional contours obtained from a plurality of real space objects, among the feature points obtained by the same method as the calculation of the operating point 3040, the uppermost feature in the row direction in the acquired image Select the 3D contour to which the point belongs. In the method of selecting one real space object for extracting operation points, in addition to this, the area of the three-dimensional curved surface formed by the three-dimensional contour, the position of the center of gravity, the number of vertices of the three-dimensional contour, and the vertex group Various features may be used, such as the average position of .

In step S2400, operation point calculation section 1040 calculates the three-dimensional position of the operation point, that is, the three-dimensional position in a coordinate system based on the position and orientation of the imaging section that captured the image that acquired the corresponding points. Based on the position and orientation, it is transformed into a three-dimensional position in the world coordinate system.

In step S2500, measurement section 1030 uses the left-eye image to obtain the position and orientation of the left viewpoint in the world coordinate system, and uses the right-eye image to obtain the position and orientation of the right viewpoint in the world coordinate system. Then, the CG generation unit 1080 uses the drawing data of each virtual object held in the holding unit 1060 to construct a virtual object and place it in the virtual space.

At that time, the CG generation unit 1080 places a pointer at the three-dimensional position of the operating point in the world coordinate system. A virtual object used as a pointer is not limited to a specific virtual object. Furthermore, the CG generation unit 1080 arranges the above buttons in the virtual space with a prescribed position and orientation. The CG generation unit 1080 generates an image of the virtual space viewed from the left viewpoint (virtual space image for the left eye), and generates an image of the virtual space viewed from the right viewpoint (virtual space image for the right eye). do.

In step S2600, synthesizing section 1090 generates, as a left-eye mixed reality space image, a synthesized image in which a left-eye virtual space image is superimposed on a left-eye image. Further, the synthesizing unit 1090 generates a synthetic image obtained by superimposing the virtual space image for the right eye on the image for the right eye as the mixed reality space image for the right eye. Thereby, for example, as shown in FIG. 3D, a pointer 3050 can be superimposed on the position of the corresponding point 3040 on the image 3000.

In step S<b>2700 , synthesizing section 1090 outputs the left-eye mixed reality space image to the left-eye display section of display section 1120 of HMD 1100 , and transmits the right-eye mixed reality space image to HMD 1100 . It is sent to the right-eye display unit of the display unit 1120 .

Next, contact determination processing between an operation point and a button will be described with reference to the flowchart of FIG. The processing according to the flowchart of FIG. 4 may be performed as part of the processing according to the flowchart of FIG. 2, or may be performed as a separate thread.

In step S4100, signal generation section 1050 determines whether or not the pointer placed at the three-dimensional position of the operation point has come into contact with the button. Various methods can be applied to the contact determination method between the pointer and the button. For example, as shown in FIG. 5, when a line segment connecting the three-dimensional position of the pointer 3050 in the previous frame and the three-dimensional position of the pointer 3050 in the current frame intersects with any of the polygons forming the button 5100, the pointer It is determined that 3050 and button 5100 have come into contact. In addition, when a three-dimensional area including the position of the pointer 3050 is set, and a plane or line segment making up the three-dimensional area intersects with any of the polygons making up the button 5100, the pointer 3050 and the button 5100 is determined to have come into contact with

As a result of such determination, if it is determined that the pointer and the button have come into contact with each other, the process proceeds to step S4200; otherwise, the process returns to step S4100.

In step S4200, signal generating section 1050 generates a signal for executing a process preset as a process to be executed when a button is pressed, and sends the generated signal to signal processing section 1070. . In step S 4300 , signal processing section 1070 executes processing corresponding to the signal received from signal generating section 1050 .

[Second embodiment]
Differences from the first embodiment, including this embodiment, will be described below, and the same as the first embodiment unless otherwise specified. Although the number of operation points to be selected is one in the first embodiment, the number of operation points to be selected may be two or more. By setting the number of selectable operation points to two or more, and making it possible to execute processing according to the position and movement of each operation point for each operation point, the type of operation by the target object can be changed from the first embodiment. can also be increased. In this embodiment, in the flowchart of FIG. 2, the process according to the flowchart of FIG. 6 is performed instead of the above steps S2100 to S2500.

In step S6100, contour generation section 1020 extracts the image area of the target object on each of the left-eye image and the right-eye image included in the stereo image. For example, as shown in FIG. 7A, image regions of hands 3010a and 3010b are extracted from an image 7000 including human hands 3010a and 3010b as target objects. As for the method of extracting the hand image area, an image area having pre-registered hand color information is extracted in the same manner as in step S2100 described above.

In step S6200, contour generation section 1020 performs labeling processing on each pixel if there are a plurality of pixels having hand color information. If the pixels extracted as pixels having hand color information in step S6100 are adjacent to each other, the same label is given to these pixels. By performing the labeling process on the image 7000 of FIG. 7A, each pixel belonging to the image area 7010 of the hand 3010a is given the label A, and the image area of the hand 3010b is assigned as shown in FIG. 7B. Each pixel belonging to 7020 is given a label B different from the label A. FIG.

In step S6300, contour generator 1020 extracts the two-dimensional contour lines of the image regions for each labeled image region in the same manner as in step S2100 described above.

In step S6400, contour generation section 1020 performs the same processing as in step S2200 described above to generate a two-dimensional contour line of the image region of interest in one image and an image region corresponding to the image region of interest in the other image. and two-dimensional contour lines are associated with each other. Then, for each set of associated two-dimensional contour lines, contour generation section 1020 performs the same processing as in step S2200 using the two-dimensional contour lines belonging to the set, thereby generating the target object corresponding to the set. Obtain a point group that constitutes the three-dimensional contour of .

In step S6500, for each pair, operating point calculation section 1040 selects, as an operating point, a point that satisfies a specified condition from the point group obtained in step S6400 for that pair, in the same manner as in step S2300. . FIG. 7C shows corresponding points 3040 selected as operation points from the point group forming the three-dimensional contour 3030 corresponding to the image region 7010, and points forming the three-dimensional contour 3031 corresponding to the image region 7020. A point 7030, selected as an operating point of the group, is shown.

In step S6600, operating point calculation section 1040 converts the three-dimensional position of each operating point into a three-dimensional position in the world coordinate system by performing the same processing as in step S2400 above for each operating point.

In step S6700, operating point calculation section 1040 ensures continuity by associating the operating points of the same target object between the operating points in the previous frame and the operating points in the current frame. For example, the distance between the three-dimensional position of each operating point obtained for the previous frame and the three-dimensional position of each operating point obtained for the current frame is obtained. Then, the operating point of interest in the current frame is associated with the operating point at the three-dimensional position closest to the operating point of interest among the operating points in the previous frame. Prediction information such as velocity information or a Kalman filter may be used as a method of association.

In step S6800, the same processing as in step S2500 is performed, but CG generation section 1080 arranges a pointer at the three-dimensional position of the operation point in the world coordinate system for each operation point. As a result, for example, as shown in FIG. 7D, the pointer 3050 can be superimposed on the position of the corresponding point 3040 on the image 3000, and the pointer 7040 can be superimposed on the position of the point 7030 as the operation point.

Next, the processing performed according to the positions of the two pointers will be described with reference to the flowchart of FIG. Note that the processing according to the flowchart of FIG. 8 may be performed as part of the above processing, or may be performed as a separate thread.

In step S8100, signal generating section 1050 satisfies the condition that two pointers are arranged in the virtual space and the distance between the two pointers is within a prescribed distance continuously for a prescribed time or longer. Determine whether or not As a result of this determination, if this condition is satisfied, the process proceeds to step S8200; otherwise, the process returns to step S8100.

In step S8200, signal generation section 1050 turns on the mode for expanding the menu panel (menu panel expansion mode). If the above condition is not satisfied, the signal generator 1050 turns off the menu panel expansion mode. When the menu panel deployment mode is turned on, the mixed reality space image is a rectangular object having two pointers, a pointer 3050 and a pointer 7040, at the ends of the diagonal line, as shown in FIG. 9A. 9100 is superimposed. Object 9100 is an object indicating that the menu panel is being expanded.

However, when the menu panel expansion mode is turned ON, signal generation section 1050 generates a signal instructing generation of object 9100 and sends it to signal processing section 1070 . Upon receiving such a signal, the signal processing section 1070 instructs the CG generation section 1080 to generate a rectangular object 9100 . Note that the positional relationship between the rectangular object 9100 and the two pointers is not limited to the positional relationship shown in FIG. 9A.

In step S8300, signal generation section 1050 determines whether or not the condition that the distance between the two pointers is equal to or greater than a specified distance is satisfied. As a result of this determination, if this condition is satisfied, the process proceeds to step S8400; otherwise, the process returns to step S8100.

In step S8400, the signal generation unit 1050 generates a signal instructing to place the menu panel on a plane whose center position is the average position of the two pointers and whose normal vector is a vector parallel to the line-of-sight vector of the imaging unit. is generated and sent to the signal processing unit 1070 . Here, the line-of-sight vector of the imaging section may be either the line-of-sight vector of the right-eye imaging section or the left-eye imaging section, or the line-of-sight vector of the right-eye imaging section and the line-of-sight vector of the left-eye imaging section. It may be an average vector of line-of-sight vectors. As a result, the signal processing unit 1070, for example, as shown in FIG. 9(B), has the average position of the two pointers as the center position, and the normal vector is a vector parallel to the line-of-sight vector of the imaging unit. The CG generation unit 1080 is instructed to arrange the menu panel 9101 .

Another process performed according to the positions of the two pointers will now be described with reference to the flowchart of FIG. Note that the processing according to the flowchart of FIG. 10 may be performed as part of the above processing, or may be performed as a separate thread. In the process according to the flowchart of FIG. 10, a unique process is assigned to each operating point, and when the pointer corresponding to the operating point satisfies a prescribed condition, the process assigned to the operating point is executed. Run.

In step S10100, signal generation section 1050 sets a role for each operating point. For example, the first role is set for the operation point located in the left half of the image, and the second role different from the first role is set for the operation point located in the right half of the image. . The method of setting the role may be set by the user via the GUI, may be set in advance, or may be set according to the usage status of the HMD 1100 .

In this embodiment, as a preliminary stage of this process, it is assumed that a menu is pressed to enter this process. (operation point B) is the operation point on the side where the button is pressed. Accordingly, in step S10100, the role of operation point A is set to display a button nearby, and the role of operation point B is set to press a button.

In step S10200, signal generation section 1050 satisfies the condition that the three-dimensional positions of the two operation points are within the field of view of imaging section 1110 (left-eye imaging section and right-eye imaging section). determine whether there is As a result of this determination, if this condition is satisfied, the process proceeds to step S10300, and if this condition is not satisfied, the process returns to step S10200.

In step S 10300 , signal generating section 1050 generates a signal instructing placement of a button near the three-dimensional position of operation point A, and sends the signal to signal processing section 1070 . As a result, the signal processing unit 1070 instructs the CG generation unit 1080 to place a button near the three-dimensional position of the operation point A. Therefore, as shown in FIG. A button 11000 is arranged near the pointer 3050 corresponding to the point A. FIG.

In step S10400, signal generation section 1050 determines whether or not the pointer corresponding to operation point B has come into contact with the button in the same manner as in step S4100 described above. FIG. 11(B) shows how the pointer and the button are in contact with each other. As a result of such determination, if it is determined that the pointer and the button have come into contact with each other, the process advances to step S10500; otherwise, the process returns to step S10200.

In step S10500, signal generating section 1050 generates a signal for executing a process preset as a process to be executed when a button is pressed, and sends the generated signal to signal processing section 1070. . Signal processing section 1070 executes processing corresponding to the signal received from signal generating section 1050 .

In this embodiment, two hands are used as target objects for which operation points can be set. However, the number of target objects for which operation points can be set may be two or more. In that case, the number of selected operating points is also two or more.

[Third Embodiment]
In the first and second embodiments, one operation point is set for one target object, but the number of operation points that can be set for one target object may be two or more. For example, when using a plurality of fingers than when using a single finger, more diverse operations can be realized. It is significant to execute processing according to the positions and movements of the set operating points. In this embodiment, in the flowchart of FIG. 2, the process according to the flowchart of FIG. 12 is performed instead of the above steps S2300 to S2500.

In step S12100, operating point calculation section 1040 obtains the degree of convexity (kurtosis) of a sharp point existing in the point group forming the three-dimensional contour. For example, as shown in FIG. 13A, the kurtosis of a sharp point 13200 existing in a point group 13100 forming a three-dimensional outline is obtained. A sharp point 13200 is obtained by projecting a group of points on a three-dimensional contour onto an image and performing convex hull processing, and defining the points of the obtained convex hull as common points of the projected contours. Note that the method for obtaining the sharp point is not limited to a specific method. As preprocessing for generating the convex hull, a low-pass filter may be applied to the projected contour. Next, at each point of the projected contour line, a predetermined number of points are selected on both sides of the obtained sharp point, and the (n-1)-th point from one end point to the other end point is selected. The sum of the angles formed by the n-th line segment and the n-th and (n+1)-th line segments is calculated. This value is defined as kurtosis. Note that the definition of kurtosis may be different, for example, the above kurtosis divided by the sum of the distances of the set of line segments composed of adjacent points from one end point to the other end point can be taken as the kurtosis.

In step S12200, operating point calculation section 1040 treats sharp points with kurtosis equal to or greater than the threshold as target sharp points to be processed below, and excludes sharp points with kurtosis below the threshold from the following processes. If this threshold value is set to 0, all sharp points can be set as target sharp points. Alternatively, the kurtosis may be obtained for each point on the three-dimensional contour line, and points having a threshold value or more may be set as target sharp points.

In step S12300, operating point calculation section 1040 selects two target sharp points from the target sharp point group. The selection condition for selecting two sharp points from the target sharp point group is not limited to a specific condition. In the present embodiment, among the target sharp points, a sharp point having coordinate values that are the uppermost and the second uppermost in the row direction when projected onto an image (one of the image for the left eye and the image for the right eye) are selected as the first and second operating points, respectively. For example, when a point group 13100 is obtained as shown in FIG. 13400 is selected as the second operating point. Note that the sharp point with the highest kurtosis may be selected as the first operating point, and the sharp point with the second highest kurtosis may be selected as the second operating point. to determine the operating point.

In step S12400, operating point calculation section 1040 performs the same processing as in step S2400 for each of the first operating point and the second operating point, thereby calculating the first operating point and the second operating point. Each 3D position is transformed into a 3D position in the world coordinate system.

In step S12500, the same processing as in step S2500 is performed, but CG generation section 1080 calculates the three-dimensional position of each of the first and second operating points in the world coordinate system. Place the pointer in the As a result, for example, as shown in FIG. 13B, a pointer 13310 can be superimposed on the position of the first operation point on the image, and a pointer 13410 can be superimposed on the position of the second operation point.

Then, the signal generation unit 1050 generates a trigger signal for performing processing according to the three-dimensional position and movement of each operation point and sends it to the signal processing unit 1070. The signal processing unit 1070 receives the signal The CG generation unit 1080 is controlled so as to perform corresponding processing. In addition to processing according to the position and movement of operation points, information on the position, orientation, and movement of each operation point may be used, and information on the relative positions, orientations, and movements of multiple operation points may be used. You may use the information. Furthermore, information on the contact state with other virtual objects and the relative position, orientation, and movement between each operation point and the virtual object may be used.

<Modification 1 of Third Embodiment>
In this modified example, another calculation method different from the method of calculating the plurality of operating points described in the third embodiment will be described. In this modified example, in the flowchart of FIG. 2, the processing according to the flowchart of FIG. 14 is performed instead of steps S2300 to S2500 described above.

In step S14100, operating point calculation section 1040 selects one of the sharp points on the three-dimensional contour as the first operating point. The selection condition for the first operating point is not limited to a specific selection condition, but in this embodiment the first operating point is selected in the same manner as in the third embodiment. For example, as shown in FIG. 15A, a first operating point 15200 is selected from a point group 15100 in the same manner as in the third embodiment.

In step S14200, operating point calculation section 1040 sets a ROI (region of interest) on the image (one of the left-eye image and the right-eye image) based on the first operating point. For example, as shown in FIG. 15B, each point on the three-dimensional contour is projected onto the image, and a rectangular area 15300 having the position of the first operating point 15200 as the midpoint of the upper side is set as the ROI.

In step S14300, operating point calculation section 1040 divides the contour formed by the points projected onto the image by the boundaries of the ROI. For example, as shown in FIG. 15(C), when a line segment formed by adjacent contour points straddles the boundary of the ROI, points outside the ROI are excluded and the contour is divided.

In step S14400, operating point calculation section 1040 selects operating point candidates from points on respective contour lines within the ROI. The method for selecting contour line candidates is not limited to a particular method, but in this embodiment, for each contour line in the ROI, the uppermost point in the row direction on the image among the points on the contour line is Make it a candidate for the operating point.

In step S14500, operating point calculation section 1040 determines a specified number of operating points such as a second operating point, a third operating point, . . . from the candidate operating points. Although the method of determining the operating point is not limited to a specific method, in this embodiment, among the candidates for the operating point, the second operating point, the third operating point, . A specified number of operating points are determined as follows.

In step S14600, operating point calculation section 1040 converts the three-dimensional position of each operating point into a three-dimensional position in the world coordinate system by performing the same processing as in step S2400 for each operating point.

In step S14700, the same processing as in step S2500 is performed, but CG generation section 1080 places a pointer at the three-dimensional position of each operating point in the world coordinate system.

<Modification 2 of Third Embodiment>
In each of the above embodiments and modifications, the three-dimensional outline of the target object is obtained using stereo images, but the method for obtaining the three-dimensional outline of the target object is not limited to a specific method. For example, as shown in FIG. 17, an HMD 1100 is equipped with a depth camera 17010 to capture a depth image of a target object. A depth image of the target object obtained by the depth camera 17010 is sent to the information processing apparatus 1000 . The contour generator 1020 uses the depth image from the depth camera 17010 to find the three-dimensional contour of the target object.

Further, in the above embodiments and modified examples, it is not identified at all who the operation point on the image belongs to or what kind of object it is. Therefore, for example, when the hands of a plurality of users are included in the image, it is also possible to set an operation point for each hand (or finger). In this case, it is possible to perform processing according to gestures based on the positions and movements of the hands of a plurality of users. Note that the user's hand is just an example of the target object, and by pre-holding the feature amount (color, shape, etc.) of the target object, the image area of the target object is detected from the image using this feature amount. As a result, the contour of the target object can be detected.

Further, in the above embodiments and modified examples, after obtaining the three-dimensional positions of all the points on the contour, the points satisfying the prescribed conditions among the point group on the contour are used as the operating points. However, it is also possible to determine the three-dimensional positions of the operating points after determining the points satisfying the specified conditions among the points on the contour line as the operating points.

[Fourth embodiment]
In the first embodiment, the predetermined feature that becomes the operation point does not particularly depend on the operation state, but if the feature for obtaining the operation point can be switched depending on the operation state, it is expected that the operability will be improved. .

For example, as shown in FIG. 19A, when the real space object is a hand, it is assumed that the selection of the CG model 19200 and the pressing of the CG button are often performed with the tip of the finger. Therefore, as shown in FIG. 19A, it is preferable to set the feature point having the uppermost coordinate value in the visual field of the HMD as the operating point 19100 . On the other hand, since the orientation and angle of the hand change during operation of the CG model, rather than setting the top point of the hand in the field of view of the HMD as the operation point, the center of gravity of the hand region is set as shown in FIG. Alternatively, setting the average center to the operating point 19400 reduces unintended behavior of the operating point. In this embodiment, a method of switching features for obtaining an operation point according to an operation state will be described.

A series of processes for switching the feature that becomes the operation point depending on the operation state will be described with reference to the flowchart of FIG. 18 .

Steps S18100-S18200 are the same as steps S2100-S2200 of the first embodiment, so description thereof will be omitted.

In step S18300, the current application state is obtained, and the subsequent processing branches depending on the state. In the present embodiment, the application state is described below using the CG model selection mode and the CG model movement mode as examples, but the application state is not limited to this.

In step S18400, operation point selection processing is executed when the application is in the CG model selection mode. In this embodiment, when the vertices of the three-dimensional contour obtained in step S18200 are projected onto the image obtained in step S18100, the vertex on the three-dimensional contour having the uppermost coordinate value in the Row direction is used as the operating point. do. The feature for calculating the operating point is not limited to this method, but it should be a method different from the method used when the application state in step S18500 is in another mode.

In step S18500, operation point selection processing is executed when the application state is the CG model movement mode. In this embodiment, the center of gravity of the three-dimensional curved surface area formed by the three-dimensional outline obtained in step S18200 is calculated and used as the operating point. The feature for calculating the operating point is not limited to this, and may be the center position of the vertex of the three-dimensional contour.

Steps S18600-S18700 are the same as steps S2400-S2500 of the first embodiment, so description thereof will be omitted.

[Fifth Embodiment]
In this embodiment, when there are a plurality of real space objects from which operation points are to be obtained, a method of selecting one real space object for extracting an operation point will be described, according to an operation state.

For example, if one real space object for extracting operation points is the real space object that exists at the top in the field of view of the HMD, when the CG model is moved downward, other real space objects When the operation point is above the real space object to which the operation point belongs, the operation point moves to another real space object, which impairs the operability.

Therefore, in this embodiment, as shown in FIG. 21A, real space objects 21300 and 21200 whose operating point 21100 is at the top of the field of view of the acquired image are selected in the CG model selection state. Also, as shown in FIG. 21B, when the CG model 21400 is in a moving state, the real space object 21300 selected at the start of movement of the CG model 21400 is tracked, and the real space object 21300 is selected.

A series of processes for switching the method of selecting one real space object for extracting an operation point according to the operation state will be described with reference to the flowchart of FIG. 20 .

The processing of steps S20100-S20400 is the same as the processing of steps S6100-S6400 of the second embodiment, so description thereof will be omitted.

In step S20500, the current application state is obtained, and the subsequent processing branches depending on the state. In this embodiment, if the application state is the CG model selection mode, the process branches to step S20600, and if it is the CG model operation mode, the process branches to step S20700.

In step S20600, a process of selecting a three-dimensional contour for calculating operating points from among the plurality of three-dimensional contours obtained in step S20400 is executed. In this embodiment, when the vertices of each 3D contour are projected onto the image acquired in step S20100, the vertex on the 3D contour having the uppermost coordinate value in the Row direction is set as the comparison point. The method of taking features for calculating a comparison point is not limited to this method, and any method may be used. Next, the three-dimensional contour to which the comparison point having the uppermost coordinates in the row direction among the plurality of three-dimensional contours belongs is selected as the three-dimensional region for calculating the operating point. The selection method for calculating the operating point from the comparison points is not limited to the method described above, but must be a different method from step S20700.

In step S20700, a process of selecting a three-dimensional contour for calculating operating points from among the plurality of three-dimensional contours obtained in step S20400 is executed. In this embodiment, in the selection mode for selecting a CG model to be moved, an event is issued for transitioning to the move mode for moving the CG model by actually selecting the CG model. In the move mode, the 3D contour to which the operating point belongs when selecting the CG model is continuously tracked, and this is selected as the 3D contour for calculating the operating point. Any method may be used for 3D contour tracking, that is, for estimating the identity between processing frames. In this embodiment, the distance of the center of gravity of the three-dimensional curved surface region generated by the three-dimensional contour is the closest between the previous frame and the current frame, and is treated as the same three-dimensional contour.

In step S20800, operating points are calculated from the three-dimensional contour line selected in step S20600 or S20700. Specifically, when the selected three-dimensional contour is projected onto the captured image, the vertex on the three-dimensional contour having the uppermost coordinate value in the row direction in the captured image is set as the operating point. The feature for calculating the operating point is not limited to this method, and any method may be used. Also, as in the fourth embodiment, the feature may be switched according to the operation mode.

Steps S20900-S20110 are the same as steps S6700-S6800 of the second embodiment, so description thereof will be omitted.

[Sixth Embodiment]
Each functional unit of the information processing apparatus 1000 shown in FIGS. 1 and 17 may be implemented by hardware, but functional units other than the holding unit 1060 may be implemented by software (computer program). In the latter case, a computer device having a memory functioning as the holding unit 1060 and capable of executing this software is applicable to the information processing device 1000 described above.

A hardware configuration example of a computer device applicable to the information processing device 1000 will be described with reference to the block diagram of FIG. 16 . Note that the hardware configuration shown in FIG. 16 is merely an example of the hardware configuration of a computer device that can be applied to the information processing device 1000 .

CPU 16200 executes various processes using computer programs and data stored in ROM 16300 and RAM 16400 . As a result, the CPU 16200 controls the operation of the entire computer apparatus and executes or controls each of the processes described above as those performed by the information processing apparatus 1000 .

The ROM 16300 stores computer programs and data that do not need to be rewritten (for example, BIOS computer programs and data). RAM 16400 has an area for storing computer programs and data loaded from ROM 16300 and external storage device 16700 and data received from the outside via input I/F 16500 . Furthermore, the RAM 16400 has a work area used when the CPU 16200 executes various processes. Thus, the RAM 16400 can appropriately provide various areas.

The input I/F 16500 functions as an interface for inputting data from the outside.

The output I/F 16600 functions as an interface for outputting data to an external device. sent to.

The external storage device 16700 is a large-capacity information storage device represented by a hard disk drive. The external storage device 16700 stores an OS (Operating System), computer programs and data for causing the CPU 16200 to execute the above-described processes performed by the information processing apparatus 1000 . The computer programs stored in the external storage device 16700 include computer programs for causing the CPU 16200 to implement the functions of the functional units other than the holding unit 1060 in FIGS. In addition, the data stored in the external storage device 16700 includes various data described as held by the holding unit 1060 and known information in the above explanation. In other words, the external storage device 16700 also functions as the holding unit 1060 described above. Computer programs and data stored in the external storage device 16700 are appropriately loaded into the RAM 16400 under the control of the CPU 16200 and are processed by the CPU 16200 .

The above CPU 16200 , ROM 16300 , RAM 16400 , input I/F 16500 , output I/F 16600 and external storage device 16700 are all connected to bus 16100 .

Some or all of the embodiments and modifications described above may be combined as appropriate, and some or all of the embodiments and modifications described above may be selectively used.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

1010: Acquisition unit 1020: Contour generation unit 1030: Measurement unit 1040: Operating point calculation unit 1050: Signal generation unit 1060: Holding unit 1070: Signal processing unit 1080: CG generation unit 1090: Synthesis unit

Claims (20)

An image processing device that outputs an image to a head-mounted display device having an imaging unit,
a selection means for selecting, as an operation point, a point that satisfies a prescribed condition from among a group of points forming the contour of the target object in the captured image captured by the imaging unit;
and processing means for executing processing based on the operating point ,
The image processing apparatus , wherein the selecting means selects, as the operation point, the uppermost point in the visual field of the head-mounted display device from the point group . selection means for selecting, as operation points, points satisfying prescribed conditions from among point groups forming contours of the plurality of target objects in the captured image;
a processing means for executing processing based on the operating point;
with
The selection means is
In the case of a selection mode in which a virtual object is selected by the operating point, selecting the operating point from a group of points forming the contours of the plurality of target objects;
In the case of a movement mode in which the virtual object is moved by the operation point, the operation point is selected from a point group forming the contour of the target object to which the operation point belonged when the virtual object was selected from among the plurality of target objects. do
An image processing apparatus characterized by: 3. The image according to claim 1, wherein said processing means arranges a virtual object at said operation point, and executes corresponding processing when said virtual object comes into contact with another virtual object. processing equipment. 3. The image processing apparatus according to claim 1, wherein said processing means arranges a virtual object near said operation point. 3. The image processing apparatus according to claim 2 , wherein said processing means executes processing based on the operation point selected by said selection means for each of said plurality of target objects. 3. The image processing apparatus according to claim 2 , wherein said processing means executes, for each operation point selected by said selection means, a process peculiar to said operation point. 3. The image processing apparatus according to claim 2 , wherein said selection means selects a plurality of points satisfying prescribed conditions from said point group as said operation points. 8. The image processing apparatus according to claim 7 , wherein said selection means selects, from said point group, points whose kurtosis is equal to or greater than a threshold as said operation points. 8. The selection means selects, as the operation points, points satisfying a prescribed condition from the point group and points satisfying the prescribed condition within an area based on the point, as the operating points. The image processing device according to . extraction means for extracting the target object from the captured image;
a contour generating means for generating a three-dimensional contour of the target object;
3. The image processing apparatus according to claim 1 , wherein said selection means selects, as said operation points, points satisfying prescribed conditions from among points forming said three-dimensional outline. 11. The image processing apparatus according to claim 10 , wherein said contour generating means generates said three-dimensional contour using stereo images. 11. The image processing apparatus according to claim 10 , wherein said contour generating means generates said three-dimensional contour using an image of said target object obtained by a depth camera. Furthermore,
generating means for generating an image of a virtual object according to the position and orientation of an imaging unit that captures the captured image;
3. The image processing apparatus according to claim 2 , further comprising means for generating a composite image of said captured image and said virtual object image, and for outputting said generated composite image to a display unit. 14. The image processing apparatus according to claim 13 , wherein said generating means generates an image of said virtual object based on said operating point. 15. The image processing apparatus according to claim 14 , wherein said generating means generates an image of a pointer arranged at a three-dimensional position of said operating point. 14. The image processing apparatus according to claim 13 , wherein the display unit is a head-mounted display device. 2. The image processing apparatus according to claim 1, wherein said selection means selects an operation point based on an operation state of said target object. An image processing method performed by an image processing device that outputs an image to a head-mounted display device having an imaging unit ,
Selecting, as an operation point, a point that satisfies a prescribed condition from among a group of points forming the contour of the target object in the captured image captured by the imaging unit ;
performing a process based on the operating point ;
The image processing method , wherein, in the selection, a point located at the uppermost position in the field of view of the head-mounted display device from among the point group is selected as the operation point . An image processing method performed by an image processing device,
For a plurality of target objects in a captured image, selecting, as operation points, points satisfying prescribed conditions from point groups forming the contours of the plurality of target objects,
performing a process based on the operating point;
In said selection,
In the case of a selection mode in which a virtual object is selected by the operating point, selecting the operating point from a group of points forming the contours of the plurality of target objects;
In the case of a movement mode in which the virtual object is moved by the operation point, the operation point is selected from a point group forming the contour of the target object to which the operation point belonged when the virtual object was selected from among the plurality of target objects. An image processing method characterized by: A computer program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 17.

Images