JP6540257B2 - AR display device, AR display method, and AR display program - Google Patents

Description

  The present invention relates to an AR display device or the like that displays AR (Augmented Reality).

  In recent years, sales promotion technology using AR has been developed. For example, Patent Document 1 discloses a mechanism in which a package of a product is attached with a marker, and when a customer shoots the package of the product with a smartphone or the like, predetermined content is superimposed and displayed on a photographed image.

JP, 2012-215989, A

  By the way, when an image captured by an in-camera (a camera mounted on a surface having a display) of a smartphone is displayed on the display, the image is automatically reversed horizontally. Therefore, when content for AR display is superimposed on this image, all display information such as characters and patterns included in the content for AR display is also displayed horizontally reversed with the image, and the display information is displayed. There is a problem that it is difficult to grasp.

  The present invention has been made in view of the above-mentioned problems, and the object of the present invention is to provide an AR display device etc. capable of easily grasping display information even when an image of right and left reverse is displayed. It is.

  According to a first aspect of the present invention for achieving the above object, there is provided a storage means for storing normal data of a package and inverted data corresponding to the normal data in association with each other, and a photographing method for photographing a photographed image including the package Means for recognizing the position, angle and magnification of the package in the photographed image on the basis of the inverted data stored in the storage means, and performing geometric transformation of the normal data corresponding to the inverted data; An AR display device comprising: control means for generating a superimposed image to be displayed superimposed on the captured image; and display means for displaying the superimposed image on the captured image. According to the first aspect of the present invention, it is possible to easily grasp the display information even when the image in which the left and right is reversed is displayed.

  The package in the first invention is a three-dimensional object, and the storage means stores shape data of the package in association with the normal data and the inverted data, and the control means is based on the shape data. The position, angle, and magnification of the package in the captured image may be recognized. By this, even if the package is a three-dimensional object, it is possible to easily grasp all the display information described in the product package.

  According to a second aspect of the present invention, there is provided a computer-implemented step of storing normal data of a package in association with inverted data corresponding to the normal data, and a photographing step of photographing a photographed image including the package The position, the angle and the magnification of the package in the photographed image are recognized based on the inverted data, the geometric conversion of the normal data corresponding to the inverted data is performed, and the superimposed image is displayed superimposed on the photographed image It is an AR display method characterized by executing a control step of generating an image and a step of displaying the superimposed image so as to be superimposed on the photographed image. According to the second aspect of the invention, it is possible to easily grasp the display information even when an image which is horizontally reversed is displayed.

  According to a third aspect of the present invention, there is provided a storage means for storing a computer, normal data of a package and inverted data corresponding to the normal data, storing the same, imaging means for capturing a photographed image including the package, and the storage A position, an angle and a magnification of the package in the photographed image are recognized based on the inverted data stored in the means, geometrical transformation of the normal data corresponding to the inverted data is performed, and the photographed image is superimposed A control unit that generates a superimposed image to be displayed and an AR display program that causes the superimposed image to function as a display unit that is superimposed on the captured image. The AR display device of the first invention can be obtained by installing the AR display program of the third invention on a general-purpose computer.

  According to the present invention, it is possible to provide an AR display device or the like capable of easily grasping display information even when a horizontally reversed image is displayed.

Hardware configuration of AR display Diagram showing an example of data Flow chart showing the flow of processing A diagram showing an example of a photographed image A diagram showing a display example of a photographed image and a superimposed image

  Hereinafter, embodiments of the present invention will be described in detail based on the drawings. The AR display device is used, for example, in real stores such as supermarkets, convenience stores, drug stores and the like. The goods are, for example, daily necessities, foods, medicines and the like. The shape of the package is, for example, a rectangular parallelepiped, a cylinder, a truncated cone, a plane or the like. Any content is only an illustration and is not particularly limited.

  FIG. 1 is a hardware block diagram of the AR display device. The hardware configuration of FIG. 1 is an example. The AR display device 1 is, for example, a smartphone or a tablet terminal, but can adopt various configurations according to the application and purpose, and may be a desktop PC (Personal Computer), a notebook PC, or the like.

  As shown in FIG. 1, in the AR display device 1, a control unit 11, a storage unit 12, an input unit 13, a display unit 14, a communication control unit 15, a camera 16 and the like are connected via a bus 17.

  The control unit 11 is configured of a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and the like. The CPU calls a program stored in the storage unit 12, the ROM, the recording medium, etc. to a work memory area on the RAM and executes it, controls driving of each device connected via the bus 17, and the AR display device 1 Implement the processing to be described later. The ROM is a non-volatile memory and permanently holds a computer boot program, a program such as a BIOS, data and the like. The RAM is a volatile memory, and temporarily stores programs, data, etc. loaded from the storage unit 12, ROM, recording medium, etc., and has a work area used by the control unit 11 to perform various processes.

  The storage unit 12 is a flash memory, a hard disk drive (HDD), or the like, and stores programs executed by the control unit 11, data necessary for program execution, an operating system (OS), and the like. As programs, a control program corresponding to the OS and an application program for causing a computer to execute processing to be described later are stored. These program codes are read by the control unit 11 as necessary, transferred to the RAM, read by the CPU, and executed as various means.

  The input unit 13 inputs data, and includes, for example, an input device such as a touch panel, a keyboard, a pointing device such as a mouse, and a numeric keypad. The user can perform an operation instruction, an operation instruction, data input, and the like to the AR display device 1 through the input unit 13. The display unit 14 includes a display device such as a liquid crystal panel, and a logic circuit (video adapter or the like) for realizing a video function of a computer in cooperation with the display device. The input unit 13 and the display unit 14 may be integrated as a touch panel display.

  The communication control unit 15 has a communication control device, a communication port, and the like, is a communication interface that mediates communication between a computer and a network, and performs communication control with another computer via the network. The network may be wired or wireless.

The camera 16 is a CMOS (Complementary Metal Oxide)
It has a Semiconductor Image Sensor), etc., and captures still images and moving images. The camera 16 is mounted on the side on which the display unit 14 is located and on both sides of the back side. A camera mounted on the side where the display unit 24 is located is called an "in-camera". The bus 17 is a path that mediates the exchange of control signals, data signals, and the like between the devices.

  FIG. 2 is a diagram illustrating an example of data. The storage unit 12 of the AR display device 1 includes shape data 21 which is a 3D model of a package, normal data 22 which is image data of each side, and image data in which normal data 22 is horizontally reversed for each package of goods. And the inverted data 23, which are

  The example of data shown in FIG. 2 corresponds to the package of goods illustrated in FIGS. 4 and 5 described later. The shape data 21 is a rectangular parallelepiped 3D model. The normal data 22 includes image data corresponding to each of the front view 22a, the right side view 22b, the left side view 22c,. In FIG. 2, the rear view, the plan view and the bottom view are omitted.

  The reverse data 23 is image data in which the image data corresponding to each surface of the normal data 22 is horizontally reversed, and includes a front view 23a, a right side view 23b, a left side view 23c,. In FIG. 2, the rear view, the plan view and the bottom view are omitted.

  Although not shown, shape data 21, normal data 22 and inverted data 23 may be stored in the storage unit 12 of the AR display device 1 for packages of a plurality of products.

  When the package is not a three-dimensional object (three-dimensional) but a plane (two-dimensional), the shape data 21 is unnecessary, and the storage unit 12 of the AR display device 1 is single for normal data 22 and inverted data 23 Only image data corresponding to the plane of the image may be stored.

  FIG. 3 is a flowchart showing the flow of processing. When the user directs the lens of the camera 16 to the package of the product and instructs the photographing, the camera 16 of the AR display device 1 photographs the package and acquires a photographed image (step S1). When shooting with the in-camera, the captured image is an image that is horizontally reversed.

  FIG. 4 is a diagram showing an example of a photographed image. In FIG. 4, the photographed image 31 is displayed on the display unit 14 of the AR display device 1. The photographed image 31 is a photograph in a state in which the user holds a package of apple juice made of a paper pack by hand.

  It returns to the explanation of FIG. The control unit 11 of the AR display device 1 recognizes the position, the angle, and the magnification of the package in the captured image 31 based on the reverse data stored in the storage unit 12 (Step S2). Specifically, the control unit 11 calculates the position, angle, and magnification of the package on the captured image 31 using software of AR (Augmented Reality: Augmented Reality). Examples of AR software include, for example, "Vuforia" (registered trademark) of QUALCOM (registered trademark), and the like. According to “Vuforia” (registered trademark), the package of the product to be targeted is a package in which the shape data 21 and the reverse data 23 shown in FIG. Position, angle, and magnification can be recognized. When data is stored in the storage unit 12 for a plurality of product packages, the control unit 11 finds out the design pattern of the package from the photographed image 31 and based on the correspondence between the shape of the package and the design made into a database. Recognize the position, angle and magnification of the package. Here, the design pattern is a pattern identified by part or all of design elements (characters, figures, symbols, patterns, colors, etc., their positions, sizes, ranges, etc.) included in the package.

  Next, the control unit 11 of the AR display device 1 performs geometric conversion of the normal data 22 corresponding to the inverted data 23 in accordance with the position, angle, and magnification of the package recognized in step S2, and converts the converted data. The normal data 22 is generated as a superimposed image (step S3). Here, geometric transformation includes linear transformation, scaling, rotation, translation, affine transformation, projective transformation and the like. The superimposed image is an image that has been geometrically transformed according to the shape of the package included in the captured image.

  In step S3, for the front view, the back view, the plan view and the bottom view of the reverse data 23, the control unit 11 geometrically relates to the front view, the back view, the plan view and the bottom view of the normal data 22, respectively. Convert and generate a superimposed image. On the other hand, the control unit 11 performs a geometrical transformation of the right side view of the normal data 22 for the right side view of the reverse data 23 and the left side view of the normal data 22 for the left side view of the reverse data 23. Perform and generate a superimposed image.

  Next, the control unit 11 of the AR display device 1 displays the photographed image and the superimposed image on the display unit 14 (step S4).

  The control unit 11 of the AR display device 1 executes the processing of step S2 to step S4 each time an image is photographed if it is a still image, and if it is a moving image, any one of the position, angle and magnification of the package. Execute every time

  FIG. 5 is a view showing a display example of a photographed image and a superimposed image. In the example illustrated in FIG. 5, superimposed images 32 a and 32 b are displayed so as to be superimposed on the captured image 31. The superimposed image 32 a is an image obtained by geometrically converting the front view 22 a of the normal data 22. The superimposed image 32 b is an image obtained by geometrically converting the left side view 22 c of the normal data 22.

  In step S3, the control unit 11 geometrically displays the right side view of the normal data 22 for the right side view of the reverse data 23, and the right side view of the normal data 22 for the left side view of the reverse data 23. Since conversion is performed to generate a superimposed image, it is possible to correctly display the left and right positional relationship in the product package in the image obtained by combining the photographed image and the superimposed image. For example, even if there is a pattern straddling all of the front view, the right side view and the left side view, it can be displayed without a sense of incongruity.

  According to the AR display device 1 in the embodiment of the present invention, it is possible to easily grasp the display information even when the image in which the left and right is reversed is displayed. In particular, since a place hidden by the user's hand or the like is accurately displayed on the captured image 31, the convenience is higher than that of the normal image inversion technology.

  Furthermore, by preparing an image different from the package of the actual product as normal data 22 on which the superimposed image is based, it is possible to provide the user with information with high added value. For example, for a user who does not understand Japanese, preparing an image in which the characters are written in a foreign language can give a surprise as if the characters of the package were automatically translated. In addition, for example, by preparing an image in which information on advertisement that a seller of a product wants to display is described, it is possible to obtain the same effect as a POP (Point of purchase advertising) advertisement of a product shelf.

  The preferred embodiments of the AR display device and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is apparent that those skilled in the art can conceive of various modifications or alterations within the scope of the technical idea disclosed in the present application, and of course these also fall within the technical scope of the present invention. It is understood.

1 ...... AR display device 21 ...... Shape data 22 ...... Normal data 23 ...... Reversed data 31 ...... Photographed image 32a, 32b ...... 重 畳 Superimposed image

Claims (4)

A storage unit that stores normal data of a package and inverted data corresponding to the normal data in association with each other;
A photographing unit for photographing a photographed image including the package;
The position, angle, and magnification of the package in the photographed image are recognized based on the inverted data stored in the storage means, and geometric conversion of the normal data corresponding to the inverted data is performed, and the photographed image Control means for generating a superimposed image to be displayed superimposed on the
Display means for displaying the superimposed image superimposed on the photographed image;
AR display device characterized by having. The package is a three-dimensional object,
The storage means stores shape data of the package in association with the normal data and the inverted data,
The AR display device according to claim 1, wherein the control unit recognizes a position, an angle, and a magnification of the package in the photographed image based on the shape data. The computer is
Storing the normal data of the package and the inverted data corresponding to the normal data in association with each other;
A photographing step of photographing a photographed image including the package;
The position, angle, and magnification of the package in the captured image are recognized based on the stored inverted data, geometrical conversion of the normal data corresponding to the inverted data is performed, and the superimposed data is displayed on the captured image. A control step of generating a superimposed image to be
Displaying the superimposed image superimposed on the photographed image;
AR display method characterized by performing. Computer,
A storage unit that stores normal data of a package and inverted data corresponding to the normal data in association with each other;
A photographing unit for photographing a photographed image including the package;
The position, angle, and magnification of the package in the photographed image are recognized based on the inverted data stored in the storage means, and geometric conversion of the normal data corresponding to the inverted data is performed, and the photographed image Control means for generating a superimposed image to be displayed superimposed on the
An AR display program for causing the superimposed image to be displayed as superimposed on the photographed image.

Images