JP5612929B2 - GUIDANCE INFORMATION PROVIDING SYSTEM, CONTROL DEVICE, GUIDANCE INFORMATION PROVIDING METHOD, AND PROGRAM - Google Patents

Description

  The present invention relates to a guidance information providing system that performs appropriate guidance display according to an instruction operation on a paper medium.

Conventionally, a navigation system that guides route information to a destination on a map has been developed. For example, some in-vehicle navigation systems measure the current position using GPS (Global Positioning System) or the like, and perform navigation by displaying directions and arrows on a map. A portable navigation device as shown in Patent Document 1 has also been proposed. In the portable navigation device shown in Patent Document 1, the map is displayed by matching the orientation of the device body with the actual orientation.
Moreover, what provides a user with a various method besides the path information displayed on a map is proposed. For example, in Patent Document 2, when the current position or destination information of an operating device is acquired in accordance with an operation from an operating device (cart) that moves within the facility, route information to the destination is generated and A guidance information providing system that outputs guidance information to a nearby output device is described. Further, in Patent Document 3, when a direction indicator is installed at an intersection in a route guidance area or a branch point of a passage in a station, and the direction indicator senses the approach of a portable terminal held by a pedestrian, Describes a route guidance system that acquires route information searched by a guidance center device based on destination coordinates held by and displays the route information on a direction indicator.

  By the way, in recent years, a technique called AR (Augmented Reality) has been developed. In AR, a marker having a predetermined image pattern is detected by a camera, and a corresponding virtual object is synthesized and displayed in a live-action image while tracking the movement of the detected marker, whereby a real space and a virtual object are displayed. It is possible to create a special video that fuses Regarding marker recognition in AR, for example, research shown in Non-Patent Document 1, Patent Document 4, and the like has been performed. Non-Patent Document 1 describes a marker position detection method and a calibration method thereof, and Patent Document 4 describes a reference marker, an object (painting), and an AR technique in order to use the AR technique more flexibly for simulation image generation. A method for normalizing is described.

Japanese Patent Laid-Open No. 7-280583 Japanese Patent Laid-Open No. 2004-347459 JP 2002-54945 A JP 2009-271822 A

Hirokazu Kato and three others, "Augmented Reality System Based on Marker Tracking and its Calibration", Journal, Virtual Reality Society of Japan, 1999, Vol. 4, No. 4

  However, all of the above-described techniques related to route guidance require a user to have an operation device such as a portable terminal or a cart. For example, when performing route guidance using the method described in the above-mentioned patent document or the like in route guidance of an exhibition, it is necessary to provide a mobile terminal or a mobile device to visitors, and in some cases The attendees had to master the operation of these devices. Further, from the standpoint of an operator of an exhibition or amusement facility, it has been desired to provide guidance information in a manner that attracts the interest of visitors.

  The present invention has been made in view of such problems, and the object of the present invention is to provide guidance that provides an augmented reality guidance in an intuitive manner using a familiar medium such as a paper medium. It is to provide an information provision system.

In order to solve the above-described problem, the first invention includes an imaging unit that acquires a paper medium possessed by a user as image information, a display unit that displays information content in an area visible from the user's viewpoint, A guidance information providing system comprising: a storage unit that stores an image pattern of a plurality of markers and information content corresponding to each marker; and a control device that controls the imaging unit and the display unit. An image recognizing unit for recognizing a marker instructed by the user on the paper medium based on image information captured by the image capturing unit, and a response from the storage unit according to the marker recognized by the image recognizing unit. information reads the content, and a display control means for displaying on said display means, said image recognition means, imaging is by the imaging means for A first determination unit that determines whether or not all the markers printed on the paper medium are contained in the image, and the imaging after the first determination unit determines that all the markers are contained Second determination means for determining whether or not there is an unrecognized marker among all the markers printed on the paper medium in the image picked up again by the means, and not recognized as determined by the second determination means Instruction marker determining means for determining a marker as a marker instructed by the user;
A guide information providing system characterized by comprising:

  Here, the marker means a print content of a paper medium having a pre-registered image pattern. The marker includes the entire design surrounded by a frame and the design itself within the frame. The design includes graphics, illustrations, and designed characters.

  According to the guidance information providing system of the first invention, when the paper medium possessed by the user is acquired as image information by the imaging unit, the image recognition unit of the control device captures the image information captured by the imaging unit. And the display control means reads out the information content corresponding to the recognized marker from the storage means, and the display means reads the information content by the user on the paper medium. To be displayed in an area that can be seen from the viewpoint.

  As a result, when the user points one of the printed contents of the paper medium held by the user with a finger or the like, the appropriate information content corresponding to the pointed marker is displayed in an area visible from the user's viewpoint. become. Therefore, the guide information can be provided in an intuitive manner using a paper medium that is a medium that can be easily provided to the user. In addition, it is possible to provide interesting information with augmented reality that guidance information (information content) is displayed in a real space, even though it is an instruction operation to ordinary paper which is a non-electronic medium. become.

In the first invention, the storage means stores a table representing the relationship between the marker and the paper surface of the paper medium, and the image recognition means is based on image information captured by the imaging means. It is desirable that the paper surface of the paper medium possessed by the user can be identified.
Here, the meaning of the term “identification of the paper surface of the paper medium” includes both the identification of the “type” of the paper medium and the identification of “page (front / back, etc.)” of the paper medium.
Thereby, the image recognition means can identify the paper surface (the type of paper medium or the page of the paper medium) from the marker included in the image information imaged by the imaging means by referring to the table.

In the guidance information providing system according to the first aspect of the present invention, the system further includes a projecting unit that performs projection on the paper medium, and the display control unit is configured such that the image recognition unit identifies the paper surface of the paper medium when All marker specifying means for projecting spot content by the projection means for all markers printed on a paper medium, and when any of the markers is recognized by the image recognition means It is preferable that the apparatus further comprises indication marker specifying means for projecting spot content onto the marker by the projection means.
The user can intuitively recognize an instructable image (marker) among the print contents of the paper medium by the spot content projected by the all marker specifying means. In addition, it is possible to intuitively recognize that the device has recognized the instructed marker from the spot content projected by the instructing marker specifying means. Therefore, it is possible to provide a guidance information providing system that is easy for the user to use.

  Here, the spot content projected by the projection unit may be not only light that illuminates the projection target locally, but also the spot content itself may indicate a shape, information, or the like.

In the guidance information providing system according to the first aspect of the present invention, it is preferable that the display means sets a display area as an area including a floor surface around a space photographed by the imaging means.
As a result, the information content appears to be projected from the viewpoint of the user so as to extend from the paper to the floor while keeping the posture in which the user is holding the paper medium. As a result, the user can naturally see the information content expanded and displayed in a wider range than the paper he has.

In the guidance information providing system according to the first aspect, when the second determination unit determines that the plurality of markers are not recognized, the instruction marker determination unit determines a priority order among the plurality of unrecognized markers. It is desirable to determine a marker having a high value as a marker instructed by the user .

2nd invention is a control apparatus provided with the memory | storage means which relates and memorize | stores the image content of a some marker and the information content corresponding to each marker, Comprising: The imaging means which images the paper medium which a user possesses Image recognition means for recognizing a marker instructed on the paper by the user on the basis of image information captured by the user, and information content corresponding from the storage means in accordance with the user-designated marker recognized by the image recognition means Display control means for displaying the corresponding information content on display means for displaying the information content in an area visible from the user's viewpoint, and the image recognition means is imaged by the imaging means First determination means for determining whether or not all markers printed on the paper medium are contained in an image; and the first determination means After it is determined that all the markers are contained, it is determined whether there is an unrecognized marker among all the markers printed on the paper medium in the image captured again by the imaging unit. And a pointing marker determining unit that determines the unrecognized marker determined by the second determining unit as a marker instructed by the user .

  According to the control device of the second invention, similarly to the first invention, when the paper medium possessed by the user is acquired as image information by the imaging unit, the image recognition unit is imaged by the imaging unit. Based on the image information, the marker designated by the user on the paper surface is recognized, and the display control means reads out the information content corresponding to the recognized marker from the storage device and displays it in an area that is visible from the user's viewpoint. .

  Accordingly, when any marker is pointed with a finger or the like on the paper medium held by the user, appropriate information content corresponding to the pointed marker is displayed in an area visible from the user's viewpoint. Therefore, guidance display can be performed in an intuitive manner using a paper medium that can be easily provided to the user. In addition, in spite of the instruction operation to ordinary paper that is a non-electronic medium, it is possible to provide an interesting and interesting guidance display with augmented reality that guidance (information content) is displayed in a real space. It becomes possible.

According to a third aspect of the present invention, there is provided an imaging step of acquiring a paper medium possessed by the user as image information, and an image recognition step of recognizing a marker designated by the user on the paper medium based on the captured image information. A display control step of reading the information content corresponding to the recognized marker from the storage device storing the information content corresponding to the image patterns of the plurality of markers and displaying the information content in an area visible from the viewpoint of the user When, only contains the image recognition step includes a first determination step of determining whether all the marker printed on the paper medium in the image that has been captured is within by the imaging means, the first After it is determined in the determination step that all the markers are contained, the paper medium is marked in the image captured again by the imaging means. A second determination step for determining whether or not there is an unrecognized marker among all the markers that have been determined, and an indication marker determination for determining the unrecognized marker determined by the second determination step as a marker instructed by the user A guidance information providing method characterized by comprising: a step .

  According to the guidance information providing method of the third invention, as in the first and second inventions, when the paper medium possessed by the user is acquired as image information, the user The marker instructed in is recognized, information content corresponding to the recognized marker is read from the storage device, and the information content is displayed in an area that is visible from the user's viewpoint.

  As a result, when the user points one of the markers with a finger or the like on the possessed paper medium, appropriate information content corresponding to the pointed marker is displayed in an area visible from the user's viewpoint. Therefore, guidance display can be performed in an intuitive manner using a paper medium that is familiar to the user. In addition, in spite of the instruction operation to ordinary paper that is a non-electronic medium, it is possible to provide an interesting and interesting guidance display with augmented reality that guidance (information content) is displayed in a real space. It becomes possible.

A fourth invention is a program described in a computer-readable format, an imaging step for acquiring, as image information, a paper medium possessed by a user who has entered a predetermined space by imaging means, and imaging An image recognition step for recognizing the marker designated by the user on the paper medium based on the image information that has been performed, and a plurality of information contents corresponding to the image patterns of the plurality of markers are stored according to the recognized marker. reads the corresponding information content from the storage apparatus, seen containing a display control step of outputting the information content to the display device, wherein the image recognition step, printed on the paper medium in the image taken by the imaging means A first determination step for determining whether or not all the markers that have been received are contained, and all of the markers by the first determination step. A second determination step of determining whether or not there is an unrecognized marker among all the markers printed on the paper medium in the image re-captured by the imaging means after it is determined that And an instruction marker determination step of determining the unrecognized marker determined in the second determination step as a marker instructed by the user .

  According to the fourth invention, it is possible to cause a computer to function as a control device of the guidance information providing system of the first invention.

  ADVANTAGE OF THE INVENTION According to this invention, the guidance information provision system etc. which display the guidance with an augmented reality by an intuitive method using the paper medium which is a medium with which a user is familiar can be provided.

The figure which shows the hardware constitutions of the guidance information provision system 1 which concerns on this invention. The block diagram which shows the internal structure of the guidance information provision system 1 The figure which shows an example of the correlation table 56 of a marker and paper. The figure which shows an example of the printing content of the paper (brochure 8) utilized by this invention, and the spot content 41 projected with respect to all the markers 8a-8f The figure which shows an example of the spot content 43 projected with respect to the instruction | indication marker 8f The figure which shows an example of the display content on a floor surface (content display area 6) Projection image of paper (brochure 8) and floor (content display area 6) viewed from the user's viewpoint Sequence diagram showing the flow of processing in the guidance information providing system 1 Flow chart explaining the flow of image recognition processing

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.
First, the configuration of the guidance information providing system 1 according to the present invention will be described with reference to FIGS.

  FIG. 1 is a diagram showing a hardware configuration of a guidance information providing system 1 according to the present invention, and FIG. 2 is a block diagram showing an internal configuration of the guidance information providing system 1.

  As shown in FIGS. 1 and 2, the guidance information providing system 1 includes information content in an imaging device 2 that captures a predetermined imaging area 11 in a space where a user 7 enters and a predetermined content display area 6 such as a floor surface. Is connected to the control device 5, and the second projection device 4 that projects the spot content on the paper surface of the brochure 8 (paper medium) or projects the predetermined content near the standing position of the user 7. Configured. The imaging device 2, the first projection device 3, and the second projection device 4 are installed on the gantry 10.

  In the example of FIG. 1, the imaging device 2 and the second projection device 4 are installed above the standing position of the user 7, and the first projection device 3 is in the content display area 6 that can be viewed from the viewpoint of the user 7. Although it is installed at a position where predetermined display contents can be projected, it is not limited to this example. However, the installation position of the imaging device 2 is a distance that can secure the imaging area 11 having a size that does not hinder the movement of the user 7 while the printed content of the brochure 8 that the user 7 sees can be recognized. Is desirable.

  The imaging device 2 is a device that images the inside of a predetermined imaging area 11, and is configured by, for example, an infrared camera. The imaging area 11 is a range in which the entire pamphlet 8 viewed by the user 7 entering the installation space of the guidance information providing system 1 can be captured. In the guidance information providing system 1 according to the present embodiment, as will be described later, it is necessary to capture the printed contents of the brochure 8 in both the state where the spot content is projected onto the brochure 8 and the state where the spot content is not projected (see FIG. 4). For this reason, it is preferable to use an infrared camera capable of imaging regardless of the brightness of the imaging target.

The first projection device 3 is a device that displays predetermined content in the content display area 6 that is visible from the viewpoint of the user 7 in the installation space of the guidance information providing system 1, and is used by a projector or the like, for example. It is done. The content display area 6 is preferably a floor as shown in FIG. 1, but may include a wall surface or the like. Further, a monitor or the like installed at a position where the user 7 can visually recognize may be used instead of the first projection device 3. Moreover, it is good also as a display medium which combined the 1st projection apparatus 3 and the monitor.
How to display the information content by the first projection device 3 will be described later (see FIGS. 6 and 7).
In addition, the first projection device 3 may include a speaker or the like that emits audio information synchronized with the projection content.

  The second projection device 4 is a device that displays predetermined display contents on the paper surface of the brochure 8 held by the user 7 in the installation space of the guidance information providing system 1 and in the vicinity of the standing position of the user 7 (projection area 12). For example, a projector or the like is used. The second projection device 4 projects a predetermined spot content on the printed content of the brochure 8 according to the operation of the user 7 (all markers explicitly or instruction markers explicitly). Further, when there is no user 7 in the imaging area 11, a standing position mark, a guide message, and the like are projected on the floor surface of the projection area 12. The spot content projected from the second projection device 4 will be described later (see FIGS. 4 and 5).

The control device 5 includes, for example, a personal computer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a storage device such as a hard disk, a communication I / F, a peripheral device I / F, and the like. Etc.
The first projection device 3, the second projection device 4, the imaging device 2, and the like are connected to the control device 5 through a peripheral device I / F.

  As shown in FIG. 2, the CPU of the control device 5 functions as an image recognition unit 51 and a display control unit 53. The ROM, RAM, or storage device of the control device 5 functions as a marker DB (database) 55 and a content DB (database) 57. The marker DB 55 and the content DB 57 may be stored in advance in the storage device of the control device 5, or may be stored in a predetermined storage medium or the like and read by the peripheral device I / F.

  The image recognition unit 51 performs processing for recognizing the content of the image captured by the imaging device 2. Specifically, the image recognition unit 51 identifies the paper surface from the printed content of the brochure 8 imaged by the imaging device 2, analyzes the marker IDs and arrangements of the markers 8a to 8f printed on the paper surface, and Identification information (paper ID) and position and orientation information are generated. The position and orientation information of the brochure 8 is information representing the orientation of the user 7 and the state of holding the brochure 8. In order to improve marker recognition and recognition accuracy, Japanese Unexamined Patent Application Publication No. 2009-271822 and “Augmented Reality System Based on Marker Tracking and Its Calibration” (Hirokazu Kato, three others, Virtual Reality Society of Japan) A well-known method as described in Journals Vol.4, No.4, 1999) may be used.

When identifying the brochure 8 (identification of the paper ID), the image recognition unit 51 extracts a marker corresponding to the image pattern of the marker registered in the marker DB 55 from the acquired captured image, and extracts each of the extracted markers. Each marker ID is specified. Further, the image recognition unit 51 identifies the page ID of the brochure 8 based on the marker ID pattern (which marker exists) included in the brochure 8. The association between the paper surface of the brochure 8 and the marker is registered in advance in the association table 56 between the marker and the paper surface shown in FIG.
The meaning of the word “identification of the brochure 8” includes both the identification of the “type” of the brochure 8 and the identification of “page (front, back, etc.)” of the brochure 8.

  The marker DB 55 stores an image pattern of a plurality of markers and an association table 56 of markers and paper shown in FIG. For example, the pamphlet 8 with the page ID “1” includes markers with marker IDs “A”, “B”, and “C”. Further, the pamphlet 8 with the page ID “2” includes markers with the marker IDs “D”, “E”, and “F”. In the case of the example in FIG. 3, when the markers with the marker IDs “A”, “B”, and “C” are included in the captured image, the image recognition unit 51 determines that the page ID is “1” and the marker ID “ When the markers “D”, “E”, and “F” are included in the captured image, the page ID is determined to be “2”.

  Further, a method of determining the paper surface ID based on the positional relationship between the plurality of marker IDs can be considered. For example, pamphlets 8 of the same exhibition, but having different languages, are printed so that the markers are arranged differently while using the same plurality of markers in each language in order to identify language differences. Shall. The image recognition unit 51 identifies the page ID based on the type of marker included, and further identifies the language from the positional relationship of each marker.

  Further, the image recognizing unit 51 is in a state where the marker 7 is pointed by the user 7 when any of the markers that should be present in the captured image is not recognized after the page ID is recognized. recognize. That is, the marker hidden by the user 7 is recognized as an instruction marker. When a plurality of markers are hidden, the instruction marker is determined based on the priority order based on the marker arrangement.

  That is, when the image recognizing unit 51 recognizes all the markers printed on the paper surface, the image recognition unit 51 numbers the marker IDs based on the arrangement on the paper surface. For example, in the pamphlet 8 shown in FIG. 4, “1”, “2”, “3” in order from the left for the upper markers 8a, 8b, 8c, and “4” in order from the left for the lower markers 8d, 8e, 8f. , “5”, “6”. When two or more markers are hidden, the marker ID having the smallest number among the hidden markers is determined as the instruction marker. For example, when the markers 8a and 8d numbered “1” and “4” are hidden, the marker 8a having the smallest number “1” is set as the instruction marker. Similarly, when the markers 8b, 8e, and 8f numbered “2”, “5”, and “6” are hidden, the smallest “2” marker 8b is set as the instruction marker. In this example, it is assumed that the user 7 is right-handed, and the instruction marker is determined as the upper priority and the left priority.

  The display control unit 53 sends appropriate projection information (information content and spot content) to the first projection device 3 and the second projection device 4 based on the marker information recognized by the image recognition unit 51 and the position and orientation information on the paper surface. The process of outputting and displaying in an appropriate position is executed.

Specifically, as shown in FIG. 4, the display control unit 53 determines that all the markers 8 a to 8 f printed on the pamphlet 8 are in the stage where the image recognition unit 51 has identified the page ID of the pamphlet 8. Then, the spot content 41 is projected by the second projection device 4 (all marker specifying means).
FIG. 4 shows a state in which the spot content 41 is projected on the markers 8a, 8b, 8c, 8d, 8e, and 8f of the pamphlet 8 by the second projection device 4. The inside of the dotted line 41 in the figure is clearly shown by the second projection device 4. The spot content 41 clearly indicates that each marker 8a, 8b, 8c, 8d, 8e, and 8f exists and the position of each marker, so that the user 7 recognizes that the specified portion can be pointed. it can. Further, it is possible to recognize that the apparatus is in a state where the paper surface is identified and can be pointed.

Further, as shown in FIG. 5, the display control unit 53 causes the second projection device 4 to project the spot content 43 on the indication marker 8f when the image recognition unit 51 recognizes the indication marker 8f (indication marker indication). means).
FIG. 5 shows a state where the spot content 43 is projected onto the marker 8 f of the brochure 8 by the second projection device 4. The spot content 43 allows the user 7 to recognize that the device side has recognized the portion (marker 8f) pointed to by the finger.

  In addition, in the example of FIG.4 and FIG.5, although the spot content 41 and 43 is the light which illuminates each marker locally, it is not limited to this. For example, the spot content itself may indicate some information or shape.

  Further, as shown in FIG. 6, the display control unit 53 causes the first projection device 3 to display appropriate information content in the content display area 6 in accordance with the instruction marker recognized by the image recognition unit 51. The information content displayed by the first projection device 3 and a table for associating the information content with the marker are stored in advance in the content DB 57. For example, when the marker ID “A” is instructed, the display control unit 53 specifies the content associated with the marker ID “A” from the association table of the content DB 57, reads the content, and requests the first projection device 3 to perform projection. To do. The first projection device 3 projects information content under the control of the control device 5.

As shown in FIG. 6, the information content projected on the content display area 6 such as a floor surface includes a destination mark 63 and a destination in addition to the content 61 expressed by moving images, still images, texts, and the like. Route 65, means for reaching the destination (foot mark 62 when walking, those marks when using stairs, elevators, etc.), distance 64 to the destination, and the like.
Note that audio information may be reproduced in synchronization with the reproduction of the content 61. In this case, a voice device (not shown) is added to the guidance information providing system 1.

Further, the display control unit 53 may control the display position of the information content or the spot content so as to follow the position of the marker designated by the user 7.
For example, when viewing the content display area 6 from the viewpoint of the user 7, the information contents 61 to 65 are continuously displayed in the content display area 6 from the position and holding angle of the brochure 8. That is, the image recognizing unit 51 executes image recognition processing for each frame image of the captured moving image, and when an instruction marker is detected from each frame image, the movement direction and movement amount of the instruction marker are tracked, and tracking information is obtained. The data is output to the display control unit 53. The display control unit 53 changes the display angle and display position of the content according to the tracking information.

  As shown in FIG. 7, from the viewpoint of the user 7, the information contents 61 to 64 are viewed as if they are expanded and displayed from the paper surface of the brochure 8 on the floor. Further, as indicated by a dotted line 45 in FIG. 7, the trajectory content 45 for continuously displaying the instruction marker 8 f of the brochure 8 and the display content of the content display area 6 is projected on the paper surface by the second projection device 4. It may be.

  The pamphlet 8 is a paper medium on which arbitrary contents are printed as shown in FIGS. 4 and 5 and the like, regardless of the material of paper or ink. However, it is a material that can obtain a contrast capable of identifying the marker from the infrared camera (imaging device 2). In particular, it is effective to use mat paper or the like on the paper surface. The printed content of the brochure 8 includes one or a plurality of markers.

  The markers include not only the entire design surrounded by a frame of a predetermined shape but also a design not surrounded by the frame, as shown in 8a to 8f of FIGS. The design includes graphics, illustrations, designed characters, and the like. The marker image pattern is registered in advance in the marker DB 55.

  Further, the marker printed on the brochure 8 may include a marker for identifying a language. Further, if there is no restriction on the paper surface design or the like, a marker for recognizing the position and orientation of the paper surface may be printed at a predetermined position on the paper surface. The print contents other than the marker are arbitrary.

Next, the operation of the guidance information providing system 1 will be described.
FIG. 8 is a sequence diagram for explaining the flow of the operation of each part of the guidance information providing system 1 performed in response to the operation of the user 7.

  As shown in FIG. 8, when the user 7 first enters the imaging area 11 with the brochure 8 (step U1) and holds the brochure 8 over the imaging area 11 (step U2), the imaging apparatus 2 The inside brochure 8 is photographed. In order to improve the imaging accuracy, the mark may be projected on the standing position by the second projection device 4 or the like. The operation of the user 7 holding the pamphlet may be such that the pamphlet 8 is held at a position or posture where the imaging apparatus 2 can capture the entire paper surface of the pamphlet 8.

The imaging device 2 captures an image of the brochure 8 held by the user 7 (held by hand) and sends the imaging information to the image recognition unit 51 of the control device 5 (step S1). The image recognizing unit 51 calculates the paper surface identification and position / orientation state of the brochure 8 based on the imaging information of the imaging device 2 (step S2). The image recognition unit 51 specifies the type of the brochure 8 and the printed marker based on the identification information and position / orientation information of the brochure 8, and requests to display the corresponding projection information (step S3). In response to the request from the image recognition unit 51, the display control unit 53 reads out the corresponding projection information from the content DB 57 and causes the corresponding projection devices 3 and 4 to project the projection information (step S4). The second projection device 4 projects predetermined projection information (including the spot contents 41 and 43, the locus contents 45, etc.) on the paper surface of the brochure 8 according to the control information sent from the display control unit 53 (step S5). Further, the first projection device 3 projects the output information contents 61 to 65 onto the content display area 6 such as a floor surface in accordance with the control information sent from the display control unit 53 (step S6).
The user 7 can see the projection information on the pamphlet 8 (step U3) and can see the projection information on the floor (content display area 6) (step U4).

  The state in which the user 7 points to a desired marker on the brochure 8 (step U5) is also captured by the imaging device 2 (step S1), and the imaging information is recognized and calculated by the image recognition unit 51 (step S2). ). Based on the recognized information, the corresponding projection information is projected by the corresponding projection devices 3 and 4 (steps S3 to S6).

  Here, with reference to FIG. 9, the image recognition process performed in the image recognition part 51 is demonstrated.

  The image recognition unit 51 captures an image captured by the imaging device 2 (step S21) and searches for a marker in the image (step S22). If the marker does not exist (step S23; No), it is determined that the paper surface is not in the correct position (step S24), and “0” is set to the paper surface ID (step S25). In this case, the image recognizing unit 51 performs processing of the next image frame (step S27) without requesting the projection of the content (step S26).

  If there is at least one marker in the captured image (step S23; Yes), the image recognition unit 51 determines the currently set paper surface ID (step S28). When the currently set page ID is “0”, all markers associated with each page ID are referred to by referring to the association table 56 between the markers registered in the marker DB 55. It is determined whether or not it exists (step S29). If all markers associated with any page ID do not exist (step S29; No), it is determined that the page is not in the correct position (step S24), and “0” is set in the page ID. (Step S25). In this case, the next image frame is processed (step S27) without requesting the projection of the content (step S26).

When there is one or more markers in the image (step S23; Yes) and the currently set paper ID is “0” (step S28; Yes), the image is linked to any paper ID in the image. If it is determined that all markers are present (step S29; Yes), it is determined that the paper surface is at the correct position (step S30), and the corresponding paper surface ID is changed (step S31). At this stage, the image recognition unit 51 requests the display control unit 53 to display the spot content 41 when the marker is not selected on the paper (step S32). At this time, the display control unit 53 may be requested to display the information content when the marker is not selected in the content display area 6 such as the floor surface.
A spot content 41 as shown in FIG. 4 is projected on each marker 8 a to 8 f on the paper surface of the brochure 8. When no marker is selected, the spot content 41 is projected onto all the markers 8a to 8f on the paper surface.

If it is determined in step S28 that the currently set page ID is not “0”, the image recognition unit 51 determines whether all markers associated with the set page ID exist in the image. Is determined (step S33).
If all the markers associated with the set paper ID exist in the image (step S33; YES), it is determined that the user 7 is not pointing at the marker (step S34), and the same as step S32 Display state. That is, the image recognition unit 51 requests the display control unit 53 to display the spot content 41 when the marker is not selected on the paper, and displays the information content when the marker is not selected in the content display area 6 such as the floor surface. The display control unit 53 is requested to do so (step S32).

  If it is determined in step S28 that the currently set page ID is not “0” and some markers associated with the set page ID do not exist in the image (step S33; NO) ) Determines that the user 7 is pointing the marker (step S35). If a plurality of markers are not recognized in step S35, a marker having a higher priority (for example, upper-level priority and left-priority) among the unrecognized markers is determined as an instruction marker (step S36).

At this stage, the image recognizing unit 51 requests the display control unit 53 to display the spot content 43 at the time of marker selection on the sheet, and also displays information content corresponding to the selected marker in a content display area such as a floor surface. 6 is requested to be displayed on the display control unit 53 (step S37).
A spot content 43 as shown in FIG. 5 is projected on the paper surface of the brochure 8. When the marker is selected, the spot content 43 is projected only on the selected marker 8f.

When the user 7 leaves the imaging area 11 (step U6 in FIG. 8), no paper surface information is captured in the imaging area 11 of the imaging device 2 (step S7). In this case, the image recognizing unit 51 specifies that the user 7 has left the imaging area 11 (step S8), and requests the display control unit 53 for projection information corresponding to the user leaving (step S9).
In response to the request from the image recognition unit 51, the display control unit 53 reads out the corresponding projection information from the content DB 57 and causes the corresponding projection devices 3 and 4 to project the projection information (step S10). For example, the first projection device 3 projects predetermined information content on the content display area 6 such as a floor surface in accordance with the control information sent from the display control unit 53 (step S11).
Next, the user 7 who has entered can view projection information such as the floor surface (step U7).

  As described above, the guidance information display system 1 according to the present embodiment includes the imaging device 2, the first projection device 3, the second projection device 4, and the control device 5. The pamphlet 8 being viewed is imaged, and the marker printed on the paper surface of the pamphlet 8 is extracted from the imaging information by the image recognition unit 51 of the control device 5. The image recognition unit 51 recognizes the marker instructed on the paper by the user 7, and the display control unit 53 displays appropriate information content corresponding to the recognized instruction marker in the information content display area 6 such as the floor surface. To display.

  In addition, when the image recognition unit 51 identifies the page of the brochure 8, the display control unit 53 and the second projection device 4 cause the spot content 41 to be projected on all the markers, and the image recognition unit 51 recognizes the instruction marker. At this stage, the display control unit 53 and the second projection device 4 cause the spot content 43 to be projected onto the indication marker.

  As a result, when the user 7 points out one of the printed contents of the brochure 8 held by the user 7 with a finger or the like, an appropriate information content corresponding to the pointed marker can be viewed from the viewpoint of the user 7 (content Displayed in the display area 6). Therefore, guidance information can be provided in an intuitive manner using a paper medium that can be easily provided to the user 7 and is a familiar medium. In addition, it is possible to provide interesting information with augmented reality that guidance information (information content) is displayed in a real space, even though it is an instruction operation to ordinary paper which is a non-electronic medium. become.

  In addition, since the spot content 41 is projected on all the markers that can be designated at the stage when the image recognition unit 51 recognizes the paper surface, the user 7 can find out the portion (marker) that can be designated in the printed contents of the brochure 8. It can be recognized intuitively. Further, when the image recognizing unit 51 recognizes the marker designated by the user 7, the spot content 43 is projected only on the designated marker, so that the user 7 can intuitively recognize that the apparatus has recognized the designated marker. Therefore, the guidance information providing system 1 that is easy for the user 7 to use can be provided.

  Further, if the information content display area 6 is an area including the floor surface around the standing position of the user 7, the user 7 keeps the posture of holding the pamphlet 8 and from the viewpoint of the user 7 from the paper. Information content appears to be projected to extend to the floor. As a result, the user 7 can naturally view the information content expanded and displayed in a wider range than paper.

  Further, the image recognition unit 51 of the control device 5 determines whether or not all the markers printed on the brochure 8 are included in the image captured by the imaging unit, and determines that all the markers are included. If it is determined that there is an unrecognized marker among all the markers, and if it is determined that there is an unrecognized marker, the marker designated by the user 7 is determined based on the arrangement of the unrecognized marker. To do. Accordingly, the image recognition unit 51 can recognize the marker instructed by the user 7 from the captured image.

In addition, although the guidance information display system 1 of this Embodiment provided the imaging device 2 in one place and imaged the imaging area 11, the installation location of the imaging device 2 may be plural. A plurality of installation positions and installation locations of the projection apparatuses 3 and 4 may be provided.
When a plurality of imaging areas 11 and a plurality of display areas 6 are provided, the brochure 8 is imaged by the imaging area 11 that can be imaged according to the standing position of the user 7, the marker pointed at is recognized, and the optimum display area 6 to display. Further, the content display area 6 may be not only in front of the user 7 but also around the user 7. In this case, the first projection device 3 is installed at a position where it can be projected onto the set display area 6. Further, the information content may be displayed by combining the first projection device and the monitor.
Moreover, the pointing operation to the paper surface by the user 7 is not limited to the case of using the finger, and the instruction marker may be hidden using an object.
In addition, it is obvious that those skilled in the art can come up with various changes and modifications within the scope of the technical idea disclosed in the present application, and these naturally belong to the technical scope of the present invention. It is understood.

DESCRIPTION OF SYMBOLS 1 ... Guidance information provision system 2 ... Imaging device 3 ... 1st projector 4 ... 2nd projector 41 ... Spot content at the time of marker non-selection 43 ... Spot at the time of marker selection Content 45 ... Trajectory content 5 ... Control device 51 ... Image recognition unit 53 ... Display control unit 55 ... Marker DB
57 ... Contents DB
6 ... content information display area 7 ... user 8 ... pamphlet 8a, 8b, 8c, 8d, 8e, 8f ... marker 10 ... mount 11 ... imaging area

Claims (8)

Imaging means for acquiring paper media possessed by the user as image information;
Display means for displaying information content in an area visible from the user's viewpoint;
Storage means for storing an image pattern of a plurality of markers and information content corresponding to each marker;
A control device for controlling the imaging means and the display means;
A guidance information providing system comprising:
The control device includes:
Image recognition means for recognizing a marker instructed by the user on the paper medium based on image information picked up by the image pickup means;
Display control means for reading the corresponding information content from the storage means according to the marker recognized by the image recognition means, and displaying the information content on the display means;
Equipped with a,
The image recognition means includes
First determination means for determining whether or not all the markers printed on the paper medium are contained in an image captured by the imaging means;
Whether or not there is an unrecognized marker among all the markers printed on the paper medium in the image re-captured by the imaging unit after it is determined that all the markers are contained by the first determination unit Second determination means for determining whether or not
Instruction marker determining means for determining an unrecognized marker determined by the second determination means as a marker instructed by the user;
A guide information providing system comprising: The instruction marker determining means includes
The plurality of markers not determined to be recognized by the second determination means, a marker having a higher priority among the plurality of unrecognized markers is determined as the marker instructed by the user. The guide information providing system according to 1.   The storage means stores a table representing the relationship between the marker and the paper surface of the paper medium, and the image recognition means is a paper possessed by the user based on the image information captured by the imaging means. The guidance information providing system according to claim 1, wherein the page of the medium can be identified. A projection unit for performing projection on the paper medium;
The display control means includes
All marker indication means for projecting spot content by the projection means for all markers printed on the paper medium at the stage when the paper surface of the paper medium is identified by the image recognition means;
An indication marker specifying means for projecting spot content by the projection means to the indicated marker at the stage when any one of the markers is recognized by the image recognition means;
The guide information providing system according to claim 3 , further comprising:   The guidance information providing system according to claim 1, wherein the display unit uses a region including a floor surface around a space photographed by the imaging unit as a display region. A control device comprising storage means for associating and storing image patterns of a plurality of markers and information content corresponding to each marker,
Image recognition means for recognizing a marker instructed on the paper by the user based on image information picked up by an image pickup means for picking up a paper medium possessed by the user;
The corresponding information content is displayed on the display means for reading the corresponding information content from the storage means according to the user-designated marker recognized by the image recognition means and displaying the information content in an area visible from the user's viewpoint. Display control means for displaying
Equipped with a,
The image recognition means includes
First determination means for determining whether or not all the markers printed on the paper medium are contained in an image captured by the imaging means;
Whether or not there is an unrecognized marker among all the markers printed on the paper medium in the image re-captured by the imaging unit after it is determined that all the markers are contained by the first determination unit Second determination means for determining whether or not
Instruction marker determining means for determining an unrecognized marker determined by the second determination means as a marker instructed by the user;
A control device comprising: An imaging step of acquiring a paper medium possessed by the user as image information;
An image recognition step for recognizing a marker instructed by the user on the paper medium based on the imaged image information;
A display control step of reading information content corresponding to the recognized marker from a storage device storing information content corresponding to an image pattern of a plurality of markers, and displaying the information content in an area visible from the viewpoint of the user; ,
Only including,
The image recognition step includes
A first determination step of determining whether or not all markers printed on the paper medium are contained in an image captured by the imaging means;
Whether or not there is an unrecognized marker among all the markers printed on the paper medium in the image re-captured by the imaging means after it is determined that all the markers are contained in the first determination step. A second determination step for determining whether or not
An instruction marker determination step for determining an unrecognized marker determined in the second determination step as a marker instructed by the user;
A guide information providing method comprising: A program written in a computer-readable format,
An imaging step of acquiring, as image information, a paper medium possessed by the user who entered the predetermined space by the imaging means;
An image recognition step for recognizing a marker instructed by the user on the paper medium based on the imaged image information;
A display control step of reading out corresponding information content from a storage device storing a plurality of information contents corresponding to the image patterns of the plurality of markers in accordance with the recognized marker, and outputting the information content to a display device;
Only including,
The image recognition step includes
A first determination step of determining whether or not all markers printed on the paper medium are contained in an image captured by the imaging means;
Whether or not there is an unrecognized marker among all the markers printed on the paper medium in the image re-captured by the imaging unit after it is determined that all the markers are contained in the first determination step. A second determination step for determining whether or not
An instruction marker determination step for determining an unrecognized marker determined in the second determination step as a marker instructed by the user;
A program for causing a computer to execute processing including

Images