JP5904379B2 - Augmented reality system, augmented reality processing method, program, and recording medium - Google Patents

Description

  The present invention uses an AR marker obtained from a captured image captured by a camera, finds a corresponding virtual image from among virtual images registered from the outside by an image transmission device, and transmits this to the camera side, The present invention relates to a completely new AR technique in which a virtual image is superimposed on the photographed image on the camera side. The virtual image registration is accepted by the virtual image server, the virtual image registration request is made by the first user-side image transmission device, and the virtual image is displayed on the second image by the second user. These are AR display devices on the user side, both of which are connected to the network.

  A technique for superimposing and displaying a virtual three-dimensional image on the display of actual view information taken by a camera is called augmented reality (AR) or augmented reality. In general, in this AR technology, an identifier such as a one-dimensional code or a two-dimensional code called an AR marker is used, and the camera captures a real field of view including the AR marker, and is associated with the AR marker. A three-dimensional image is searched for, and this virtual three-dimensional image is superimposed on the photographed real field of view information and displayed.

  The product information providing system disclosed in Japanese Patent Application Laid-Open No. 2014-041508 uses the AR technology described above, and includes product information relating to a product including a mobile terminal carried by a user and a server and having a one-dimensional barcode attached thereto. In the product information providing system, the mobile terminal reads the barcode of the product included in the image captured by the imaging unit and the imaging unit and transmits the barcode to the server. A communication unit that receives product information related to the product corresponding to the barcode transmitted from the server; and a display unit that displays the received product information superimposed on an image captured by the imaging unit. Receives a database storing product information corresponding to the barcode and a barcode transmitted from the mobile terminal, while receiving the received barcode. A communication unit for transmitting to the portable terminal the product information corresponding to the de-reading from the database, but with a.

JP 2014-041508 A

  In the AR technology used in Japanese Patent Application Laid-Open No. 2014-041508, product information corresponding to the product image received from the server is superimposed and displayed on the product image captured by the user (user) by the imaging unit of the mobile terminal. That is, it is important to display AR.

However, there are many such technologies, and Japanese Patent Application Laid-Open No. 2014-041508 is representative, but the relationship that the user side receives the product information unilaterally with respect to the server side that provides the product information. It is in. That product information be those that were prepared on the server side, the user side is not is also given any of the decision-making authority for to this.

Meanwhile, in this type services such as JP 2014-041508, the user type is equal, that is different for example two users of purposes to use the same server is not assumed. In other words, there has never been an AR technology that consists of a server, a user who only receives the service provided by the server, and a user who is actively involved in the service provided by the server. It was.

  Therefore, the present invention provides (a) that the server side can be involved in preparing the product information to be displayed on the AR, (b) a user who only receives the service provided by the server, It is an object to make one or both of the solutions to enable the AR technology to be configured with three users who are actively involved in the service contents provided by the server.

  Therefore, as a result of diligent research, the inventors have come up with the idea that an image to be a virtual image for AR display may be received from the image transmission device on the user side and registered in advance in the virtual image storage device. Got. The server provides the user with an image that is a virtual image.

  In other words, the above problem is that the virtual image server receives an image to be a virtual image to be described later from an image transmission device via a network such as the Internet, registers it in the virtual image storage device in advance, and adds an AR marker photographed by the camera. The AR marker of the printed medium is received via the network, the virtual image such as a moving image or a still image linked to the AR marker is found, the virtual image is read from the virtual image storage device, and the virtual image is read. This is achieved by transmitting to the AR display device so that the AR image is provided on the AR display device provided with the camera.

Augmented reality or augmented reality is to display a virtual image superimposed on the display of real field of view information captured by a camera. This virtual image is transmitted from the user's image transmission device to the virtual image storage device. The technology registered in advance is completely new to the present invention. The virtual image data may or may not include audio data. As the AR display device, a personal computer, a so-called Internet-capable television receiver, a smartphone, a tablet terminal, a wearable computer, or the like can be freely set. The camera may be standard equipment or external.

  When a user shoots a print medium with an AR marker with the camera of the AR display device and transmits this video to the virtual image server, the virtual image server extracts the AR marker and associates it with the AR marker. The virtual image is determined and read from the virtual images recorded in the virtual image storage device, and the virtual image is transmitted to the AR display device. Then, in the AR display device, the virtual image received from the virtual image server is displayed in AR so as to be superimposed on the print medium with the AR marker taken by the camera. In fact, this virtual image is what the user himself / herself transmitted from the image transmission device to the virtual image server and registered in the virtual image storage device. Users no longer receive product information unilaterally. Therefore, the above problem (a) has been achieved.

  When a print medium is photographed with a camera, the photograph is a still image, but it may become a moving image by video photography. In any case, an AR marker is included in the image of the print medium taken. For example, when the AR marker is a QR code (registered trademark, the same applies hereinafter), the virtual image server extracts the QR code from the video and finds a virtual image associated with the QR code value. I do. Alternatively, the video itself of the print medium is used as an AR marker, and a video matching the video itself is searched from a plurality of videos and a virtual image associated with the searched video is found. A method of recognizing characters using a word or phrase as an AR marker may be used. The technology around this can be freely selected from the prior art. Alternatively, if there is a technology to be developed in the future, this may be adopted.

  Therefore, any AR marker can be used as long as it can be a marker such as a character string, an image, a QR code, and a chameleon code (registered trademark), and various products such as rice can be used as a print medium. Anything can be used as long as it can be attached with an AR marker, such as a package, a greeting card or a T-shirt. For example, consider the following services. That is, for example, two types of users having different purposes of use with respect to the virtual image server are assumed, and these are defined as a first user and a second user. Suppose the first user has a child and receives a celebration from the second user. Therefore, the first user gives rice to the second user as a gift for the birthing celebration. A photo of the newborn is printed on the rice packaging bag as a print medium. Therefore, the second user holds the smartphone camera on which the application software called the AR display device is installed and holds it over the photo. A video is displayed. In this service, the photo itself is used as an AR marker. When newborn photo data is sent to the virtual image server, is there the same photo data among the photo data registered in the virtual image server in advance by the first user? If it is retrieved and searched, the moving image of the newborn associated with the photograph data is read from the virtual image storage device and transmitted to the AR display device of the second user. In this way, a new AR that has never existed between the server, the second user who only receives the service provided by the server, and the first user who is actively involved in the service content provided by the server. The service can be easily configured. Therefore, the above problem (b) has been achieved.

  Note that the above (2) does not apply only to the field of gifts. For example, a combination of a poster that announces an event and a video shot of the event venue, or a combination of a music CD jacket and a video shot at a concert. The industry of this invention can also be used for a wide range of applications such as the combination of New Year's cards and videos taken of the family's recent situation, and the combination of pocket tissue with promotional prints and videos taken inside the store. It is proof that the contribution to development is high.

  Next, the image transmission apparatus may further transmit print information to be attached to the print medium so as to be registered in the virtual image server. In the example of the above-mentioned celebration, the first user transmits a newborn virtual moving image to the virtual image server by the image transmission device and registers it in the virtual image storage device. Therefore, the first user also sends a photo of the newborn baby, which is print information for printing on the rice packaging bag, to the virtual image server and registers it for use. In addition, this use refers to associating at least a photograph and a virtual moving image. Therefore, in the above example, the printing information is attached to the packaging bag by the first user, by the virtual image server side, by the rice seller, or otherwise. It will be good.

  In the above-mentioned example of the internal celebration, a photo of the newborn as a still image is used as print information for printing on the rice packaging bag transmitted by the first user, and this photo itself was also an AR marker. The print information may be an illustration, a brushstroke, or only a barcode as an AR marker, for example.

  Next, a gift can be used for the print medium. In the example of the above-mentioned celebration, rice is used, but the present invention is applied to so-called gift items in general. Or, in a broad sense, greeting cards can be included in gifts.

  Next, the virtual image storage device may accept an update of the image by the image transmission device. The gist of the present invention is that (a) the user himself or (b) the first user himself / herself transmits the image from the image transmission device to the virtual image server and registers it in the virtual image storage device. By making it possible to update this virtual image, that is, by revising the association with the AR marker, (b) a user or (b) a second user newly adds an AR marker with the camera of the AR display device. When a print medium marked with is photographed, it is possible to make the virtual image displayed by the AR look different from the previous one.

Next, the virtual image server can accept registration of a plurality of the images by the image transmission device and link the plurality of the images to one AR marker. By accepting multiple registrations of the image, variations in AR display can be increased. For the virtual image data, one still image, a plurality of still images, a single moving image, a plurality of moving images, etc. may be used freely. However, in the case of the moving image described in paragraph 0015, The virtual image server can select one of the plurality of moving images and transmit it to the AR display device. Then, on the AR display device side, it is possible to design such that these can be displayed in a list and the user can select an image he / she wants to see. Alternatively, it is possible to design such that these are sequentially displayed as described in paragraph 0043.

  The above augmented reality processing method can be provided as a program for causing a computer to execute the method. Moreover, it can provide as a computer-readable recording medium which recorded this program.

  In this invention, the server has the user provide an image that is a virtual image for AR display, that is, the server receives it from the user's image transmission device and registers it in the virtual image storage device in advance. Characterized by points. As a result, when the server side prepares the product information to be displayed on the AR, the user side can be involved in this, or the user who only receives the service provided by the server and the service content provided by the server are positive. As a result, it became possible to configure AR technology with three related users, and the intended purpose was achieved.

  1 is an explanatory diagram schematically illustrating a system according to a first embodiment.   1 is an explanatory diagram of a system according to Embodiment 1. FIG.   It is explanatory drawing which represented the AR display part of Example 1 with the block diagram.   It is explanatory drawing showing the moving image transmitter of Example 1 with the block diagram.   It is explanatory drawing which represented the AR process part of Example 1 with the block diagram.   FIG. 6 is an explanatory diagram schematically illustrating an operation state of the first embodiment.   It is explanatory drawing showing the process step of the moving image transmitter of Example 1. FIG.   FIG. 5 is an explanatory diagram illustrating processing steps of a virtual image registration unit according to the first embodiment.   It is explanatory drawing showing the process step of AR display part of Example 1. FIG.   It is explanatory drawing showing the process step of AR process part of Example 1. FIG.   It is explanatory drawing showing the process step of AR display part of Example 1. FIG.   It is explanatory drawing which represented the AR display part of Example 2 with the block diagram.   FIG. 10 is an explanatory diagram illustrating processing steps of a virtual image registration unit according to a third embodiment.   FIG. 10 is an explanatory diagram schematically illustrating an operation state of the fourth embodiment.

  This embodiment will be described with reference to FIGS. There are two types of users on the server side, but the second user shown in the left hand of FIG. 1 requests the server side to transmit the virtual image 32 and displays the transmitted virtual image 32 on the display 10. To do. This is the relationship between the conventional server and client. The special feature of the first embodiment is an image (in this example, a moving image) that is a virtual image 32 transmitted from the image transmission device of the first user shown in the lower part of FIG. Is registered in the virtual image storage unit 54 on the server side. Conventionally, the virtual image 32 is registered on the server side as the virtual image 32, and so on the user side as an outsider, possession, selection, registration, etc. of the image that becomes the virtual image 32 It was unthinkable.

  In this embodiment, there are two types of users on the server side. In other words, as described above, it is assumed that a child is born to the first user and the childbirth celebration is received from the second user. Therefore, the first user decides to give the second user the rice for the celebration. Giving rice here depends on the new rice coming to my home. The first user places an order for rice to the merchant belonging to the server. At this time, the photo is provided to the merchant with the new address 30 and the name of the newborn. As a result, the seller prints the newborn's photograph 30 and name on the rice bag 3 and delivers it to the destination. So far, this is a conventional service.

  In addition to such a service, the first user passes this photo 30 to the server side. As described above, the photo 3 of the newborn is printed on the rice bag 3 as the printing medium. Then, when a second user who has received rice holds the camera 11 of the smartphone 1 on which application software called an application to be the AR display device 4 is installed over the rice bag 3, the virtual image 32 is a photo 30 of the newborn. Is displayed so as to overlap the captured image 31 of the rice bag 3 including The first user is also the second user. The virtual image 32 can be reproduced by holding the camera 11 of the smartphone 1 in which the application software is installed over the photograph 30 of the rice bag 3. Hereinafter, such a mechanism will be described in detail.

  A smartphone 1 having a display 10 on the front side and a camera 11 on the back side is connected to the Internet wirelessly. For convenience of explanation, the smartphone 1 is the same as that of the first user and that of the second user, and the function of performing the AR display is the same. It has special functions as described below. A virtual image server 2 is also connected to the Internet (FIG. 2).

  When the AR display unit 4 of the smartphone 1 of the second user is represented by the block diagram of FIG. 3, the AR display unit 4 includes an imaging unit 40, a display unit 41, and a transmission / reception unit 42. The same configuration also includes the smartphone 1 of the first user, but as illustrated in FIG. 4, the virtual image 32 is registered in the virtual image storage unit 54 of the virtual image server 2 as the moving image transmission device 43. A moving image capturing unit 44 and a moving image transmitting unit 46 for capturing an image are configured. Reference numeral 45 indicates a display unit for a captured moving image.

  On the other hand, when the AR processing unit 5 of the virtual image server 2 is represented by the block diagram of FIG. 5, the AR processing unit 5 includes the search unit 50, the image storage unit 51, the image determination unit 52, the virtual image registration unit 53, and the virtual image storage unit. 54 and a transmission / reception unit 55.

  The operation of each block is as follows. That is, as shown in FIG. 7, the first user uses the smartphone 1 as the moving picture transmission device 43 and takes a picture of the newborn's moving picture while confirming the moving picture shooting section 44 of the camera 11 on the display section 45 (step S1). In order to register the moving image, the moving image transmitting unit 46 transmits the moving image to the virtual image server 2 (step S2).

  Then, as shown in FIG. 8, the virtual image server 2 receives the captured moving image 32 (what will be treated as a virtual image later) from the moving image transmitting device 43 by the transmission / reception unit 55 (step S <b> 3). Is recorded in the virtual image storage unit 54 by the virtual image registration unit 53 (step S4). The virtual image server 2 separately records the data of the photograph 3 printed on the rice bag 3 described above in the image storage unit 51, so that the photograph 3 and the captured moving image 32 are associated with each other.

  Then, as shown in FIG. 9, when the second user holds the photograph 30 of the rice bag 3 with the camera 11 of the smartphone 1 (step S5), the captured image is processed by the display unit 41 of the AR display unit 4 and displayed on the display 10. At the same time, the captured image is compressed (step S6), and transmitted to the virtual image server 2 via the Internet via the transmission / reception unit 42 (step S7).

As shown in FIG. 10, in the virtual image server 2, when the transmission / reception unit 55 of the AR processing unit 5 receives the captured image (step S <b> 8), the video determination unit 52 matches the captured image, i. Find the image from the image storage unit 51 (step S9), and if it is searched out the virtual image that is linked, namely shooting videos 32 determines in this photo 3 (step S10), and the shooting The moving image 32 is read out from the virtual image storage unit 54 and transmitted to the smartphone 1 of the second user via the transmission / reception unit 55 (step S11).

  FIG. 6 schematically shows such processing. When the image determination unit 52 receives data of the captured image (c), the images (a, b, c, d, e, f, g,..., and the image (c) is searched out. The virtual image (C) recorded in the virtual image storage unit 54 is linked to the image (c). Therefore, when it is determined that this virtual image (C) is to be used, and is read from the virtual image storage unit 54 and transmitted from the transmission / reception unit 55, the virtual image (C) is displayed on the second user's smartphone as described below. The AR 10 is displayed on the first display 10.

  That is, as shown in FIG. 11, the captured image (c) of the photograph 30 of the rice bag 3 previously captured is displayed on the display 10 of the smartphone 1 of the second user. The virtual image (C) transmitted from the AR processing unit 5 is received by the transmission / reception unit 42 (step S12), and this virtual image (C) is superimposed on the captured image (c) 31 by the display unit 41. Displayed (step S13). Therefore, it is possible to display on the photograph 30 of the rice bag 3 so that the video of the newborn baby registered in the virtual image server 2 by the user who gave the birth celebration appears. The effect of further enhancing happiness is outstanding.

  Next, similarly to the first embodiment, the smartphone used in this embodiment includes a display 10 on the front side and a camera 11 on the back side, and is connected to the Internet wirelessly. When the AR display device 6 of the smartphone is represented by the block diagram of FIG. 12, the AR display device 6 includes an AR marker reading unit 60, a display unit 61, and a transmission / reception unit 62.

  A label (not shown) on which a photo of a newborn and an AR marker 36 are printed is attached to a rice cosmetic box. Therefore, when this is photographed with the camera of the smartphone, the AR marker reading unit 60 reads the AR marker from the photographed image, and transmits the AR marker data from the transmission / reception unit 62 to the virtual image server via the Internet. In this embodiment, a QR code is used for the AR marker 36.

  On the virtual image server side, although not shown, when the transmission / reception unit of the AR processing unit receives the AR marker data, the image determination unit uses the search unit to store the marker data that matches the received AR marker data in the AR marker storage unit. The virtual image of the place searched from the inside and linked to the searched AR marker is read from the virtual image storage unit, and transmitted to the AR display unit of the smartphone by the transmission / reception unit. The virtual image at this time is registered by the moving image transmitting unit on the second user side communicating with the virtual image server.

  This embodiment will be described with reference to FIG. Example 3 is an example in which when the second user registers a virtual image in the virtual image server, the image that becomes the same virtual image is not always placed in the virtual image server but can be changed as necessary. It is.

  That is, when the AR processing unit (not shown) of the virtual image server receives a new captured video from the video transmission device of the first user by the transmission / reception unit (step S15), the virtual image update registration unit stores the captured video in the virtual image storage. The update is registered in the part (step S16). Therefore, when the second user accesses the virtual image server, even when the smartphone is held over the same rice bag, different virtual images before and after the update are displayed as ARs, and the fun is not exhausted.

  The AR processing unit of this embodiment will be described with reference to FIG. 5, not shown, but the processing of the entire system is schematically shown in FIG. The virtual image server according to the fourth embodiment is characterized in that it accepts a plurality of registrations of images to be virtual images from the moving image transmission device 43 and links these virtual images to one AR marker.

  When the image determination unit 52 receives the data of the photographed image (d), the image (d) is applied by hitting the images (a, b, c, d, e, f, g,...) Recorded in the image storage unit 51. This image (d) is associated with virtual images (D1), (D2), and (D3) recorded in the virtual image storage unit 54. Therefore, the image determination unit of this embodiment randomly selects one from the three virtual images (D1), (D2), and (D3), determines the virtual image (D2) here, and stores this as a virtual image storage. When read from the unit 54 and transmitted from the transmission / reception unit 55 to the AR display device 4, the virtual image (D 2) is displayed as an AR so as to be superimposed on the captured image (d) on the display 10 of the smartphone 1 of the second user.

In the above description, the image determination unit randomly selects one of the three virtual images (newborn video) each time the AR display device 4 is accessed. Instead, the image determination unit transmits three serially one after another. It is also possible to set so as to. Alternatively, the AR display device 4 may receive all three virtual images (newborn moving images), display them in combination, and display the three moving images continuously. It may be possible to perform.

  Although four types of embodiments have been described so far, the present invention is not limited to these embodiments, and various variations can be developed. For example, in the example of a gift, a newborn baby photo 30 is given to a seller who receives an order for rice, and is also provided to the operation side of the virtual image server. Although the seller belongs to the server side, the server side and the seller may be integrated. The photograph 30 can be provided as a printed matter or can be transmitted as print data from a print information transmitting unit.

  In the first place, the present invention starts with not only giving a gift but also making it more fun and interesting. Therefore, the AR technology is used, but in the above-described embodiment, the virtual image is substantially provided from the first user to the second user. However, he explained that the first user can also be the second user. Therefore, it is possible to configure the system so that both parties can use this service both when giving a gift and when giving it back.

  If this invention is applied, an AR marker can be attached to a non-movable object such as a building or a tree. These are also within the scope of this invention.

DESCRIPTION OF SYMBOLS 1: Smartphone 10: Display 11: Camera 2: Virtual image server 3: Rice bag 30: Photograph 31: Captured image 32: Virtual image 4: AR display device 40: Image capturing unit 41: Display unit 42: Transmission / reception unit 43: Video transmission device 44: moving image capturing unit 45: display unit 46: moving image transmission unit 5: AR processing unit 50: search unit 51: image storage unit 52: image determination unit 53: virtual image registration unit 54: virtual image storage unit 55: transmission / reception unit 6 : AR display device 60: AR marker reading unit 61: Display unit 62: Transmission / reception unit

Claims (4)

Processing a newborn photo as an AR marker attached to a package of rice for congratulation as a print medium and a moving image of a newborn baby as a virtual image linked to the AR marker; a virtual image server; An augmented reality system comprising an image transmission device on one user side, an AR display device on a second user side remote from the image transmission device, and a network for connecting them, A camera for photographing the print medium is provided, and a photographed image photographed by the camera is transmitted to the virtual image server, or the virtual image is received from the virtual image server and the virtual image is superimposed on the photographed image. It said AR display device for and displaying Te, received directly without transmitting the captured image from the AR display device to the image transmission device of the first user Heading the virtual image which is linked to the AR marker included in the captured image, and the virtual image server to be transmitted to the AR display device reads out the virtual image from the virtual image store, and the virtual image The image and the AR marker to be attached to the print medium are transmitted only to the virtual image server without being transmitted to the AR display device, and the image to be the virtual image and the AR marker are transmitted to the image server. The AR marker received at the same time as the image to be the virtual image is configured on the virtual image server side. An augmented reality system, wherein the print medium with the mark attached is sent to the second user side . A method for processing a photo of a newborn as an AR marker attached to a package of rice for congratulation as a print medium and a moving image of a newborn as a virtual image linked to the AR marker, a virtual image server, Processed by an augmented reality system comprising an image transmission device on one user side, an AR display device on the second user side remote from the image transmission device, and a network for connecting them In the augmented reality processing method, the virtual image server is addressed only to the virtual image server without being transmitted from the image transmission device to the AR display device via the network, and becomes the virtual image on the image server. At the same time, in order to clarify the association between the image and the AR marker, the transmitted image and the AR marker are received and the virtual image recording is received. The virtual image server side registers the AR marker received at the same time as the virtual image and sends the AR marker to the second user side as the print medium with the AR marker attached thereto. The virtual image server receives a captured image that is captured by the camera of the AR display device and directly transmitted without being transmitted to the image transmission device of the first user, and is included in the captured image. The virtual image associated with the AR marker is found, the virtual image is read from the virtual image storage device and transmitted to the AR display device, and the AR display device receives the virtual image and receives the virtual image. An augmented reality processing method for displaying an image superimposed on the captured image. A program for causing a computer to execute the augmented reality processing method according to claim 2 . A computer-readable recording medium on which a program for causing a computer to execute the augmented reality processing method according to claim 2 is recorded.

Images