JP6694907B2 - Judgment device, judgment method and judgment program - Google Patents

Description

  The present invention relates to a determination device, a determination method, and a determination program.

  Conventionally, there is known an AR (Augmented Reality) technique for recognizing a predetermined target from an image and superimposing and displaying the object on the image. Moreover, there are a marker AR and a markerless AR in the AR technology system.

  The marker AR is a method of using a specific recognition target such as a QR (Quick Response) code (registered trademark) created for the purpose of being recognized by a computer. On the other hand, the markerless AR is a method in which an arbitrary image set in advance is used as a recognition target without using a specific recognition target such as a QR code. Also, tools for creating contents using the markerless AR are known (for example, see Non-Patent Documents 1 and 2).

ptc, "Vuforia", [online], [Search March 6, 2018], Internet (https://developer.vuforia.com/) CYBERNET, “SmartAR”, [online], [March 6, 2018 search], Internet (http://www.cybernet.co.jp/ar-vr/products/smartar/)

  However, the conventional technique has a problem that it may be difficult to easily create content using the markerless AR. For example, when creating content using the markerless AR, it is necessary to prepare an image suitable for a recognition target in order to improve recognition accuracy. Here, when a conventional tool is used, whether or not the prepared image is suitable for the recognition target is determined after the image is loaded into the tool. Therefore, in order to prepare an image suitable for a recognition target before being imported into the tool, a certain level of knowledge or skill is required, and it is particularly difficult for a beginner to create a content using the markerless AR. May be.

  The determination device according to the embodiment includes an extraction unit, a calculation unit, and a determination unit. The extraction unit extracts a feature point from a predetermined area in the image captured by the image capturing unit that captures the image. The calculation unit calculates an angle between the image plane of the image plane and the feature plane and the horizontal plane that are virtual planes generated based on the feature points. The determination unit determines, based on the feature point and the angle, whether or not a predetermined condition regarding the availability of the markerless AR as a marker of the image of the predetermined region imaged by the imaging unit is satisfied.

FIG. 1 is a diagram for explaining the outline of the determination device according to the first embodiment. FIG. 2 is a diagram illustrating an example of the configuration of the determination device according to the first embodiment. FIG. 3 is a diagram illustrating an example of a screen of the determination device according to the first embodiment. FIG. 4 is a diagram illustrating an example of a screen of the determination device according to the first embodiment. FIG. 5 is a diagram illustrating an example of a screen of the determination device according to the first embodiment. FIG. 6 is a diagram illustrating an example of a screen of the determination device according to the first embodiment. FIG. 7 is a diagram illustrating an example of a screen of the determination device according to the first embodiment. FIG. 8 is a diagram illustrating an example of a screen of the determination device according to the first embodiment. FIG. 9 is a flowchart showing the flow of processing of the determination device according to the first embodiment.

  Hereinafter, embodiments of a determination device, a determination method, and a determination program according to the present application will be described in detail with reference to the drawings. The present invention is not limited to the embodiments described below. In the following description, the image to be recognized in the markerless AR may be referred to as “marker”.

(First embodiment)
First, the outline of the determination device according to the present embodiment will be described with reference to FIG. FIG. 1 is a diagram for explaining the outline of the determination device according to the first embodiment. As shown in FIG. 1, the determination device 10 is carried by the user 10a. For example, the determination device 10 is a tablet terminal, a smartphone or the like.

  Further, the determination device 10 can display an image captured by using the imaging unit included in the determination device 10 on the display unit 12. Here, it is assumed that the determination device 10 is provided with a camera as an imaging unit. In the example of FIG. 1, the determination device 10 images a scene around the facility 20. The imaging target of the determination device 10 is not limited to the equipment 20 as long as it includes the recognition target of the markerless AR, and may be an artificial object such as a server device, a control panel, a printer, or a natural object. You may.

  The determination device 10 performs a determination process of determining whether or not a predetermined area in the captured image can be used as a marker, and presents the determination result to the user 10a. As shown in FIG. 1, the determination device 10 receives an operation of the user 10a via each button. The determination device 10 also superimposes and displays an object for presenting information to the user 10a on the captured image.

  First, when the start button 121b is pressed, the determination device 10 starts processing. Here, the determination device 10 receives the designation of the area used as the marker from the user 10a. For example, when the user 10a designates two points on the captured image, the determination apparatus 10 sets a rectangular area having the two points as an apexes as an area to be used as a marker and displays a frame 122a indicating the area. Also, when the reset button 121a is pressed, the designated area is reset.

  Next, when the feature point extraction button 121c is pressed, the determination device 10 extracts the feature points in the area. The determination device 10 may simultaneously extract the feature points outside the region. For example, the determination device 10 extracts the edge portion of the image as a feature point. In this case, the corners and lines of the object, and the part where the difference in brightness and color is clearly shown are more likely to be extracted as the feature points. In addition, it is preferable that a plurality of feature points exist on the same plane as an image for recognition. Here, when the determination device 10 cannot extract sufficient feature points from the area, the determination apparatus 10 displays a message prompting the user to re-specify the area.

  The determination device 10 also displays the extracted feature points as points 122b. Further, when part or all of the frame 122a is out of the imaging range of the camera of the determination device 10, a message indicating that part or all of the area is not within the imaging range is displayed.

  Furthermore, the determination device 10 performs processing for determining whether or not the posture of the camera is correct. When the camera is horizontal and the imaging surface of the camera is parallel to the characteristic surface that is a virtual surface generated from the extracted characteristic points, the determination device 10 determines that the posture of the camera is correct.

  The determination device 10 calculates an angle for determining whether the camera is horizontal and the characteristic surface and the imaging surface of the camera are parallel to each other. Specifically, the determination device 10 calculates the angle between the feature plane and the horizontal plane and the imaging plane of the camera. Note that the determination device 10 can calculate the angle between the determination device 10 and the horizontal plane by using the gyro sensor. Further, the fact that the characteristic surface and the imaging surface of the camera are parallel to each other means that the characteristic surface and the optical axis of the camera are vertical.

  Then, the determination device 10 controls the capture button 121d to be operable when the calculated angle is within the predetermined range. Further, when the calculated angle is not within the predetermined range, the determination device 10 controls the capture button 121d to be inoperable, and corrects the posture of the camera so that the angle is within the range that can be regarded as horizontal or vertical. Display a prompt message.

  The determination device 10 also displays the grid 122c on the characteristic surface in an overlapping manner. The determination device 10 also displays the mark 122d on the virtual imaging surface. Further, the determination device 10 displays the mark 122e on the characteristic surface. At this time, the determination device 10 displays the mark 122d and the mark 122e so as to overlap each other when the virtual imaging surface and the characteristic surface are parallel to each other. Further, the distance between the mark 122d and the mark 122e becomes smaller as the virtual imaging surface and the characteristic surface are parallel to each other. The marks 122d and 122e are examples of the first mark and the second mark, respectively.

  When the capture button 121d is pressed, the determination device 10 acquires the image captured by the camera as the marker image. At this time, the determination device 10 further acquires an image with an expanded imaging range as an editing image. The determination device 10 may display a message instructing the user 10a to pull the camera in order to acquire an image with a wide imaging range. The marker image and the editing image are examples of the first image and the second image, respectively.

  For example, in the example of FIG. 1, the frame 122a is displayed up in the marker image, and both the equipment 20 and the object 30 are displayed in the editing image. The marker image is used to generate the marker of the markerless AR. On the other hand, the editing image is used as a mark when arranging the AR object with respect to the marker.

(Configuration of the determination device of the first embodiment)
The configuration of the determination device 10 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of the configuration of the determination device according to the first embodiment. As shown in FIG. 2, the determination device 10 includes an input unit 11, a display unit 12, an imaging unit 13, a storage unit 14, and a control unit 15.

  The input unit 11 receives input of data from the user. The input unit 11 is, for example, an input device such as a mouse or a keyboard. The display unit 12 is a display device such as a display that displays a screen. The input unit 11 and the display unit 12 may be touch panel displays. The image capturing unit 13 captures an image. For example, the imaging unit 13 is a camera.

  The storage unit 14 is a storage device such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), an optical disk, a RAM (Random Access Memory), a flash memory, and an NVSRAM (Non Volatile Static Random Access Memory). The storage unit 14 stores an OS (Operating System) and programs executed by the determination device 10. Further, the storage unit 14 stores various kinds of information used in executing the program.

  The control unit 15 controls the determination device 10. The control unit 15 is a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. For example, the control unit 15 functions as the extraction unit 151, the calculation unit 152, the determination unit 153, the setting unit 154, the display control unit 155, and the acquisition unit 156.

  The extraction unit 151 extracts feature points from a predetermined area in the image captured by the image capturing unit 13. The extraction unit 151 can extract the edge portion of the image as a feature point. Further, the extraction unit 151 may extract the feature points of the area designated by the user, or may extract the feature points of the entire area of the image.

  The calculation unit 152 calculates the angle between the image plane of the image plane of the image plane and the characteristic plane and horizontal plane that are virtual planes generated based on the characteristic points. First, the calculation unit 152 specifies the image pickup surface of the image pickup unit 13 using Euclidean restoration. Further, the calculation unit 152 identifies the characteristic surface including the region by back projection. Then, the calculation unit 152 calculates the angle between the identified characteristic surface and the imaging surface. Further, the calculation unit 152 gyro sensor can be used to calculate the angle between the horizontal plane and the imaging surface.

  The determination unit 153 determines whether or not a predetermined condition regarding the availability of the markerless AR as a marker of the image of the predetermined region imaged by the imaging unit 13 is satisfied based on the feature point and the angle. Further, the determining unit 153 determines that the predetermined condition is not satisfied when at least a part of the predetermined region set by the setting unit 154 is out of the image capturing range of the image capturing unit 13.

  Further, the determination unit 153 determines that the predetermined condition is not satisfied when the number of feature points included in the area is equal to or smaller than the predetermined threshold value or when the angle is not within the predetermined range. For example, the determination unit 153 determines that the condition is not satisfied when the number of feature points included in the region is 5 or less. Further, for example, the determination unit 153 determines that the condition is not satisfied when the angle between the characteristic surface and the imaging surface is not within the range of −2 ° to 2 °. If the angle between the horizontal plane and the imaging surface is not within the range of 88 ° to 92 °, it is determined that the condition is not satisfied.

  The setting unit 154 sets the area designated by the user as a predetermined area. The setting unit 154 can set, as a predetermined region, a rectangular region having two points designated by the user as opposite vertices in the image captured by the image capturing unit 13. The setting unit 154 may automatically set the area if not specified by the user.

  The display control unit 155 displays the image captured by the image capturing unit 13 on the display unit 12, and when the determining unit 153 determines that the predetermined condition is not satisfied, the display control unit 155 indicates that the predetermined condition is not satisfied. The message is displayed on the display unit 12. In addition, the display control unit 155 displays the frame 122a indicating the region, the grid 122c indicating the characteristic surface, and the point 122b indicating the characteristic point in an overlapping manner on the image captured by the image capturing unit 13. Further, the display control unit 155 defines the mark 122d on the virtual image pickup surface of the image pickup unit 13 and the mark 122d that is parallel to the characteristic surface and is parallel to the virtual image pickup surface and the characteristic surface. The overlapping mark e is further displayed.

  Here, each screen displayed by the display control unit 155 on the display unit 12 will be described with reference to FIGS. 3 to 7. 3 to 7 are diagrams illustrating examples of screens of the determination device according to the first embodiment. First, as shown in FIG. 3, when the start button 121b is pressed, the display control unit 155 displays a message for explaining the operation.

  Further, as shown in FIG. 4, the display control unit 155 displays a frame 122a indicating the designated area. Further, when the feature point extraction button 121c is pressed, the display control unit 155 displays the point 122b indicating the extracted feature point. At this time, if the number of feature points in the frame 122a is equal to or less than the predetermined threshold value, the display control unit 155 displays a message "feature point extraction failure, please try again."

  Further, as shown in FIG. 5, when a part or all of the frame 122a is out of the imaging range, the display control unit 155 displays a message "feature points are out of the screen."

  Further, as shown in FIG. 6, when the angle between the horizontal plane and the imaging surface is not within the predetermined range, the display control unit 155 displays a message “Please make the terminal horizontal”. Further, as shown in FIG. 7, when the angle between the characteristic surface and the imaging surface is not within the predetermined range, the display control unit 155 displays a message “Please make the terminal vertical”. Further, in the example of FIG. 7, since the relationship between the characteristic surface and the imaging surface is largely apart from parallel, the distance between the mark 122d and the mark 122e is larger than that in the example of FIG. 6 and the like.

  When the determining unit 153 determines that the predetermined condition is satisfied, the acquiring unit 156 acquires the marker image captured by the image capturing unit 13 together with the information that specifies the predetermined region. In addition, the acquisition unit 156 further acquires an editing image that is an image including a predetermined area and has a wider imaging range than the marker image.

  For example, as illustrated in FIG. 8, the acquisition unit 156 acquires an image as an editing image after the equipment 20, the frame 122a, and the object 30 are all within the imaging range. FIG. 8 is a diagram illustrating an example of a screen of the determination device according to the first embodiment. At this time, the determination device 10 may display a message prompting the user to pull the image capturing unit 13, or may change the focal length of the image capturing unit 13 so as to expand the image capturing range.

(Processing of the first embodiment)
With reference to FIG. 9, the determination device 10 sets the area in the imaged image based on the user's designation (step S101). Then, the determination device 10 extracts feature points from the set area (step S102). Here, when the feature points in the area are not sufficient (step S103, No), the determination device 10 displays a message prompting the user to re-specify the area (step S104), and returns to the processing of step S101.

  On the other hand, when the feature points in the area are sufficient (step S103, Yes), the determination device 10 determines the posture of the image capturing unit 13 based on the angle between the feature plane and the horizontal plane and the image capturing surface of the image capturing unit 13. It is determined whether is correct (step S105). Here, when the posture of the image pickup unit 13 is not correct (No in step S105), the determination device 10 displays a message prompting correction of the posture of the image pickup unit 13 (step S106), and returns to the process of step S105.

  On the other hand, when the posture of the image pickup unit 13 is correct (Yes in step S105), it is determined whether or not the area is within the photographing range of the image pickup unit 13 (step S107). Here, when the area is not within the imaging range of the imaging unit 13 (step S107, No), the determination device 10 displays a message indicating that the area is not within the imaging range of the imaging unit 13 (step S108), and the step It returns to the process of S107.

  On the other hand, when the area is within the shooting range of the image capturing unit 13 (step S107, Yes), the determination device 10 acquires an image (step S109). At this time, the determination device 10 can acquire the marker image and the editing image.

(Effects of the first embodiment)
In the embodiment, the extraction unit 151 extracts feature points from a predetermined area in the image captured by the image capturing unit 13. Further, the calculation unit 152 calculates the angle between the image plane of the image plane of the image plane and the characteristic plane and horizontal plane that are virtual planes generated based on the characteristic points. The determination unit 153 determines whether or not a predetermined condition regarding the availability of the markerless AR as a marker of the image of the predetermined region imaged by the imaging unit 13 is satisfied based on the feature point and the angle.

  In this way, the determination device 10 can automatically determine whether or not the imaged image can be used as a marker of the markerless AR. Therefore, according to the first embodiment, even a beginner can easily create contents using the markerless AR.

  The setting unit 154 sets the area designated by the user as a predetermined area. Further, the determining unit 153 determines that the predetermined condition is not satisfied when at least a part of the predetermined region set by the setting unit 154 is out of the image capturing range of the image capturing unit 13. As a result, it is possible to prevent the extracted feature points from not being included in the acquired image.

  The determination unit 153 determines that the predetermined condition is not satisfied when the number of feature points included in the area is equal to or less than the predetermined threshold value or when the angle is not within the predetermined range. This makes it possible to acquire an image that is highly useful as an image for a marker.

  The display control unit 155 displays the image captured by the image capturing unit 13 on the display unit 12, and when the determining unit 153 determines that the predetermined condition is not satisfied, the display control unit 155 indicates that the predetermined condition is not satisfied. The message is displayed on the display unit 12. This allows the user to recognize that the image being picked up is unsuitable as a marker image.

  The display control unit 155 causes the frame 122a indicating the region, the grid 122c indicating the characteristic surface, and the point 122b indicating the characteristic point to be displayed in an overlapping manner on the image captured by the image capturing unit 13. This allows the user to intuitively understand the relationship between the imaging surface and the surface on which the feature points exist.

  The display control unit 155 is a mark that is parallel to the mark 122d on the virtual image pickup surface of the image pickup unit 13 and that overlaps with the mark 122d when the virtual image pickup surface and the feature surface are parallel to each other. e and are further displayed. This allows the user to clearly recognize that the imaging surface and the surface on which the characteristic points are present face each other.

  When the determining unit 153 determines that the predetermined condition is satisfied, the acquiring unit 156 acquires the marker image captured by the image capturing unit 13 together with the information that specifies the predetermined region. This makes it possible to acquire an image that is highly useful as an image for a marker.

  The acquisition unit 156 further acquires an editing image that is an image including a predetermined area and has a wider imaging range than the marker image. This makes it possible to acquire an image for editing in addition to the image for marker.

  The setting unit 154 sets, as a predetermined region, a rectangular region having two points designated by the user as opposite vertices in the image captured by the image capturing unit 13. This allows the user to intuitively specify the area.

  The program executed by the determination device 10 of the present embodiment is provided by being pre-installed in a ROM or the like. The program executed by the determination device 10 of the present embodiment may be configured to be provided by being recorded in a computer-readable recording medium in a file in an installable format or an executable format.

  Furthermore, the program executed by the determination device 10 of the present embodiment may be stored in a computer connected to a network such as the Internet and provided by being downloaded via the network. Further, the program executed by the determination device 10 of the present embodiment may be provided or distributed via a network such as the Internet.

  The program executed by the determination device 10 of the present embodiment has a module configuration including the above-described units (extractor 151, calculator 152, determination unit 153, setting unit 154, display control unit 155, and acquisition unit 156). As the actual hardware, the CPU reads out the program from the ROM and executes the program to load the above units onto the main storage device, and the extraction unit 151, the calculation unit 152, the determination unit 153, the setting unit 154, and the display unit. The control unit 155 and the acquisition unit 156 are generated on the main storage device.

  Although the embodiments of the present invention have been described, the embodiments are presented as examples and are not intended to limit the scope of the invention. The embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. The embodiments and modifications thereof are included in the invention described in the claims and equivalents thereof, as well as included in the scope and the gist of the invention.

10 determination device 10a user 11 input unit 12 display unit 13 imaging unit 14 storage unit 15 control unit 20 equipment 30 object 121a reset button 121b start button 121c feature point extraction button 121d capture button 122a frame 122b point 122c grid 122d, 122e mark

Claims (9)

An extraction unit that extracts feature points from a predetermined area in the image captured by the image capturing unit,
A calculation unit that calculates an angle between the image plane of the image plane and a feature plane and a horizontal plane that are virtual planes generated based on the feature points,
Based on the characteristic points and the angles, it is determined whether or not a predetermined condition regarding availability as a marker of markerless AR (Augmented Reality) of an image of the predetermined region imaged by the imaging unit is satisfied. A judgment unit,
The image captured by the image capturing unit is displayed on the display unit, and the frame indicating the region, the grid indicating the characteristic surface, and the point indicating the characteristic point are superimposed on the image captured by the image capturing unit and displayed. A display control unit that causes the display unit to display a message indicating that the predetermined condition is not satisfied, when the determination unit determines that the predetermined condition is not satisfied.
A determination device having: Further comprising a setting unit for setting the area designated by the user to the predetermined area,
The determination unit determines that the predetermined condition is not satisfied when at least a part of the predetermined region set by the setting unit is out of the imaging range of the imaging unit. The determination device according to claim 1.   The determination unit determines that the predetermined condition is not satisfied when the number of the feature points included in the area is equal to or less than a predetermined threshold value or when the angle is not within a predetermined range. The determination device according to claim 1 or 2, characterized in that. The display control unit is parallel to the first mark on the virtual imaging surface of the imaging unit and the characteristic surface, and when the virtual imaging surface and the characteristic surface are parallel to each other, The determination device according to claim 1 , wherein a second mark that overlaps the first mark is further displayed. When the determining unit determines that the predetermined condition is satisfied, the image capturing unit further includes an acquiring unit that acquires the first image captured by the image capturing unit together with information that identifies the predetermined region. The determination device according to any one of claims 1 to 4 , characterized in that. The determination device according to claim 5 , wherein the acquisition unit further acquires a second image that is an image including the predetermined region and has a wider imaging range than the first image.   The said setting part sets the rectangular area | region which has two points designated by the said user as an apex in the image image | photographed by the said imaging part to the said predetermined area | region. Judgment device. A determination method executed by a computer,
An extraction step of extracting feature points from a predetermined region in the image captured by the image capturing unit,
A calculation step of calculating an angle between a feature plane and a horizontal plane that are virtual planes generated based on the feature points and the imaging plane of the imaging unit;
Based on the characteristic points and the angles, it is determined whether or not a predetermined condition regarding the availability of the image of the predetermined region captured by the image capturing unit as a marker of markerless AR (Augmented Reality) is satisfied. Judgment process,
An image captured by the image capturing unit is displayed on a display unit, and a frame indicating the region, a grid indicating the characteristic surface, and a point indicating the characteristic point are superimposed on the image captured by the image capturing unit and displayed. And a display control step of displaying a message indicating that the predetermined condition is not satisfied on the display unit, when it is determined by the determination step that the predetermined condition is not satisfied,
A determination method characterized by the above. Computer,
Extraction means for extracting feature points from a predetermined area in the image captured by the image capturing section;
Calculation means for calculating an angle between the image plane of the image plane and the feature plane and the horizontal plane that are virtual planes generated based on the feature points,
Based on the characteristic points and the angles, it is determined whether or not a predetermined condition regarding the availability of the image of the predetermined region captured by the image capturing unit as a marker of markerless AR (Augmented Reality) is satisfied. Determination means,
An image captured by the image capturing unit is displayed on a display unit, and a frame indicating the region, a grid indicating the characteristic surface, and a point indicating the characteristic point are superimposed on the image captured by the image capturing unit and displayed. And a display control means for displaying a message indicating that the predetermined condition is not satisfied on the display unit, when it is determined by the determination means that the predetermined condition is not satisfied,
A determination program characterized by the above.

Images