JP7110738B2 - Information processing device, program and information processing system - Google Patents

Description

The present invention relates to an information processing device , a program, and an information processing system.

AR (Augmented Reality) technology is used to superimpose and display various types of information on a captured image obtained by capturing an image of a real space. For example, Patent Document 1 proposes a device that superimposes a virtual object associated with a real object in a captured image on the captured image in a display mode according to the relative position or orientation between the real object and the imaging device. It is Further, Japanese Patent Laid-Open No. 2002-200001 discloses a configuration in which a virtual object associated with a real object specified by a user among real objects in a captured image is displayed with emphasis over other virtual objects.

Japanese Patent No. 6056178

In the apparatus as described above, when a virtual object is superimposed on each real object in a captured image, a large number of virtual objects may be displayed. There is Also, when a large number of virtual objects are displayed, it is difficult for the user to check all the virtual objects and select the real object (virtual object) that he/she is interested in.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and an object of the present invention is to provide an information processing apparatus capable of displaying a virtual object (display information of an object) in an easy-to-see state. That's what it is.

An information processing apparatus according to an aspect of the present invention includes an image acquisition unit that acquires an image, a detection unit that detects an object existing in the acquired image, a distance to the detected object, and user attributes. When the information specifying unit that specifies the display information of the object according to the information, the output unit that outputs the specified display information, and the detection unit detect a plurality of objects of the same or the same genre, the plurality of the objects a determination unit that determines whether or not a distance between objects is less than a predetermined distance, and the information specifying unit determines, when it is determined that a distance between the plurality of objects is less than the predetermined distance, the plurality of objects and the display information according to the user's attribute .

In one aspect of the present invention, since display information according to the distance to the object present in the image and the attribute of the user is used, the display information of the object can be displayed in an easy-to-see state according to the user. can.

1 is a block diagram showing a configuration example of an information processing apparatus according to Embodiment 1; FIG. It is a schematic diagram which shows the structural example of product information DB and attribute information DB. FIG. 4 is a schematic diagram showing a configuration example of a detail level table; 3 is a block diagram showing functions implemented by a control unit of the information processing apparatus; FIG. It is a schematic diagram which shows the example of a display screen of an information processing apparatus. 4 is a flow chart showing a procedure of processing performed by an information processing apparatus; FIG. 8 is a block diagram showing functions implemented by a control unit of the information processing apparatus according to the second embodiment; It is a schematic diagram which shows the example of a display screen of an information processing apparatus. 4 is a flowchart showing part of the procedure of processing performed by the information processing apparatus; FIG. 11 is a block diagram showing functions realized by a control unit of the information processing apparatus according to the third embodiment; 4 is a flowchart showing part of the procedure of processing performed by the information processing apparatus; FIG. 14 is a block diagram showing functions implemented by a control unit of the information processing apparatus of the fourth embodiment; It is a schematic diagram which shows the example of a display screen of an information processing apparatus. FIG. 11 is an explanatory diagram of processing performed by an information processing apparatus according to a fifth embodiment; FIG. 12 is a block diagram showing functions implemented by a control unit of the information processing apparatus of Embodiment 6; FIG. 14 is a block diagram showing a configuration example of an information processing device and a server device according to a seventh embodiment; 4 is a flow chart showing a procedure of processing performed by an information processing device and a server device; 4 is a flow chart showing a procedure of processing performed by an information processing device and a server device;

An information processing apparatus, an information processing method, a program, and an information processing system according to the present disclosure will be described in detail below based on drawings showing embodiments. In the following embodiments, a product (object) in a captured image obtained by photographing a commercial facility such as a shopping mall or an event venue is specified, and information (display information) about the specified product is specified. is superimposed on the captured image and displayed.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration example of an information processing apparatus according to the first embodiment. The information processing device 10 of this embodiment is, for example, a smart phone, a tablet terminal, a personal computer, or the like. The information processing apparatus 10 may also be a wearable device such as a spectacles type, goggles type, headgear type, or contact lens type that is worn on the user's face or head, for example. The information processing apparatus 10 includes a control section 11, a storage section 12, a camera 13, a display section 14, an input section 15, a communication section 16, etc., and these sections are interconnected via a bus.

The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), or a GPU (Graphics Processing Unit). The control unit 11 appropriately executes control programs stored in the storage unit 12 to perform various information processing, control processing, and the like that the information processing apparatus 10 should perform.

The storage unit 12 includes a RAM (Random Access Memory), flash memory, hard disk, SSD (Solid State Drive), and the like. The storage unit 12 stores in advance control programs executed by the control unit 11 and various data necessary for executing the control programs. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the control program. The control programs stored in the storage unit 12 include an information superimposition application program (hereinafter referred to as an information superimposition application) 12a, which is a program of the present disclosure. The data stored in the storage unit 12 includes a product information database (hereinafter referred to as product information DB) 12b, an attribute information database (hereinafter referred to as attribute information DB) 12c, and a detail level table 12d. The information superimposition application 12a, the product information DB 12b, the attribute information DB 12c, and the detail level table 12d are acquired from an external device via the communication section 16 and stored in the storage section 12, for example. Further, when the information processing apparatus 10 includes a reading unit or the like for reading information stored in a portable storage medium, the information superimposition application 12a, the product information DB 12b, the attribute information DB 12c, and the detail level table 12d are read from the portable storage medium. and stored in the storage unit 12. Further, the product information DB 12b, the attribute information DB 12c, and the detail level table 12d may be stored in an external storage device or the like connected to the information processing device 10, or may be stored in another storage device with which the information processing device 10 can communicate via a network. It may be stored in a storage device.

The camera (image pickup unit) 13 has a lens, an image pickup device, and the like, and acquires image data by photoelectrically converting light incident through the lens with the image pickup device. The camera 13 takes an image according to an instruction from the control unit 11 and sequentially sends acquired image data (captured image) to the control unit 11 . Note that the information processing apparatus 10 may have a configuration in which the camera 13 is built-in, or a configuration in which a connection unit to which an external camera can be connected may be provided. In this case, the connection unit accepts input of image data acquired by the external camera and sequentially sends the input image data to the control unit 11 .

The display unit 14 is, for example, a liquid crystal display or an organic EL display, and displays various information according to instructions from the control unit 11 . The input unit 15 is a touch sensor or a push-type button arranged on the surface of the display unit 14 , receives an operation input by the user, and sends a control signal corresponding to the operation content to the control unit 11 . The input unit 15, which is a touch sensor, forms a touch panel integrally with the display unit 14. FIG. The input unit 15 may have a configuration for receiving an input by a user's hand gesture or a line-of-sight input using a function of detecting a movement of the user's hand and a function of detecting the line of sight of the user. Further, the input unit 15 may have a configuration for receiving voice input using a microphone. The communication unit 16 is an interface for connecting to a network, and transmits and receives information to and from an external device via the network.

In addition to the configuration described above, the information processing apparatus 10 of the present embodiment includes a distance sensor that measures the distance from the camera 13 (information processing apparatus 10) to an object within the imaging range during imaging processing by the camera 13. may The information processing apparatus 10 of the present embodiment includes a camera 13 and a display unit 14, and any information device capable of displaying an image captured by the camera 13 on the display unit 14 can be used. It's okay. Further, the information processing apparatus 10 of this embodiment may be configured without the communication unit 16 . When the information processing apparatus 10 is configured as a wearable device, the input unit 15 is preferably configured to receive input by the user's hand gesture, voice input by the user, line-of-sight input, and the like.

FIG. 2 is a schematic diagram showing a configuration example of the product information DB 12b and the attribute information DB 12c, and FIG. 3 is a schematic diagram showing a configuration example of the detail level table 12d. In the product information DB 12b, product information related to products sold at predetermined commercial facilities is registered. The product information includes product ID, product genre, and display information. The product ID is identification information uniquely assigned to each product. The genre is, for example, the type of product, the type of brand, or the like. The display information is information to be superimposed and displayed on the captured image when each product is captured, and includes a message and icon information. The message is text data including, for example, product description information, advertisement information, etc. The icon information is information for displaying an icon of a predetermined size. The genre and display information are information set by the seller of each product when registering the product information of each product. Note that the product information may be acquired from a website via the Internet. Also, the product information may include information on the store that sells each product.

The attribute information DB 12c registers the degree of association between user attributes and each product registered in the product information DB 12b. Specifically, in the attribute information DB 12c, the user attributes including the gender and age of the user correspond to the product ID of each product and the degree of relevance set for each set of the user attribute and the product ID. It is stored with In the example shown in FIG. 2B, large, medium, and small are registered as the degree of association between each user attribute and each product in descending order, but the degree of association is not limited to three stages. For example, each degree of association may be represented by a number or the like. The degree of relevance is, for example, information set by the seller of each product, and a higher value is set for the attribute of the target person who wants to explain and advertise each product. Further, the degree of association may be automatically obtained using machine learning, deep learning, or the like from the relationship between statistical data of past product sales and attributes of purchasers. Also, the degree of relevance of a new product to be newly launched may be predicted from the degree of relevance of similar existing products by calculating the degree of similarity with existing products using machine learning, deep learning, or the like. Furthermore, the degree of association may be set using additional data such as weather and season, in which case the accuracy of the degree of association increases.

In the detail level table 12d, detail levels associated with the degree of association registered in the attribute information DB 12c and the distance from the subject to the product at the time of photographing are registered. Specifically, the detail level table 12d stores the degree of relevance, the distance to the product, and the level of detail set for each set of the degree of relevance and the distance to the product in association with each other. The detail level is information indicating the size of the display area and the degree of detail of the display content when the display information of each product is superimposed on the captured image when each product is captured. In the example shown in FIG. 3, numbers 1 to 5 are registered as the detail level in order of the display area being large and the display content being detailed. The detail level is, for example, information set by a provider who provides the information superimposition application 12a. In the example shown in FIG. 3, any one of the first distance, the second distance, and the third distance is registered as the distance to the product. For example, the first distance is less than the first threshold, the second distance is greater than or equal to the first threshold and less than the second threshold, and the third distance is greater than or equal to the second threshold. For example, 2 m can be set as the first threshold, and 4 m can be set as the second threshold. In this case, the first distance is a distance of less than 2 m, the second distance is a distance of 2 m or more and less than 4 m, and the third distance is a distance of 4 m or more. The first threshold and the second threshold may be set to appropriate distances to the subject within the range that can be imaged by the camera 13, and the first threshold is set to a shorter distance than the second threshold. Note that the distance to the product is not limited to these, and the distance steps are not limited to three steps.

Next, functions realized by the control unit 11 executing the information superimposition application 12a in the information processing apparatus 10 will be described. FIG. 4 is a block diagram showing functions realized by the control unit 11 of the information processing apparatus 10, and FIG. 5 is a schematic diagram showing an example of a display screen of the information processing apparatus 10. As shown in FIG. When the information superimposition application 12a stored in the storage unit 12 is executed, the control unit 11 of the information processing apparatus 10 acquires the image acquisition unit 101, the product detection unit 102, the degree of association identification unit 103, the distance detection unit 104, the detail level Each function of the identifying unit 105, the first reading unit 106, the converting unit 107, the first superimposing unit 108, the selection receiving unit 109, the second reading unit 110, and the second superimposing unit 111 is implemented. In this embodiment, each of these functions is realized by the control unit 11 executing the information superimposition application 12a, but some of these functions may be realized by dedicated hardware circuits.

The image acquisition unit 101 causes the camera 13 to perform imaging processing and acquires a captured image. FIG. 5A shows a display example of a captured image. Four products A to D (subjects) are shown in the captured image shown in FIG. 5A.

The product detection unit 102 detects whether or not the captured image acquired by the image acquisition unit 101 includes a product registered in the product information DB 12b. For example, the product detection unit 102 uses a pre-learned product identifier for products registered in the product information DB 12b. identify whether In addition, the product classifier has already learned the image feature amounts such as the appearance shape and color of the product registered in the product information DB 12b. Alternatively, the product detection unit 102 may identify products included in the captured image by template matching technology. In this case, a template indicating the image feature amount of each product registered in the product information DB 12b is stored in the storage unit 12 in advance. The detected area is specified as the product area. For example, in the case of the captured image shown in FIG. 5A, the product detection unit 102 identifies that products A to C are included in the captured image. Note that the subject D in the captured image shown in FIG. 5A is assumed to be a product that is not registered in the product information DB 12b, and the product detection unit 102 treats such a subject D in the same manner as the background. The product detection unit 102 identifies a product ID for each detected product (subject).

The degree-of-association identification unit 103 identifies the degree of association with the attribute of the user of the information processing apparatus 10 for each product detected (identified) by the product detection unit 102 based on the attribute information DB 12c. Specifically, the degree-of-association identification unit 103 reads, for each product, the degree of association associated with the product ID and the gender and age of the user here from the attribute information DB 12c. User attributes are registered in advance and held in the information superimposition application 12 a or stored in the storage unit 12 . User attributes may be configured to be received via the input unit 15 when the control unit 11 executes the information superimposition application 12a.

The distance detection unit 104 detects the distance from the camera 13 (information processing device 10) to each product detected by the product detection unit 102 at the time of imaging. For example, if the information processing apparatus 10 includes a distance sensor, the distance detection unit 104 detects the distance to each product by the distance sensor during imaging. Further, the distance detection unit 104 may detect the distance to each product based on the size of the imaging area of each product (subject) in the captured image.

The detail level identifying unit 105 sets the detail level associated with the degree of association identified by the degree of association identifying unit 103 and the distance detected by the distance detecting unit 104 to each product detected by the product detecting unit 102 as a detail level. It is specified based on the table 12d. Specifically, for each product, the detail level identifying unit 105 determines that the distance detected by the distance detecting unit 104 is a first distance that is less than a first threshold, a second distance that is greater than or equal to the first threshold and less than a second threshold, It is determined which of the third distances is greater than or equal to the second threshold. Then, the level-of-detail specifying unit 105 reads the level of detail associated with the degree of association and the distance (the first distance, the second distance, or the third distance) from the level-of-detail table 12d.

The first reading unit (information acquisition unit) 106 reads display information corresponding to each product detected by the product detection unit 102 from the product information DB 12b.
The converting unit 107 converts the display information read by the first reading unit 106 for each product detected by the product detecting unit 102 to a degree of detail corresponding to the level of detail specified by the level of detail specifying unit 105. converted to fit in the display area of the specified size. For example, detail levels 1 to 3 are associated with balloon-shaped display areas that display messages included in the display information of each product, and detail levels 1, 2, and 3 are associated in descending order of the display area. . Detail levels 4 and 5 are associated with icon display areas. For example, detail level 4 is associated with a display area in which icons can be displayed based on icon information included in the display information of each product. Detail level 5 is associated with a display area in which a predetermined icon can be displayed. Therefore, the conversion unit 107 converts the message included in the display information read by the first reading unit 106 to a product whose detail level is 1 to 3, for example, at a detail level corresponding to the detail level 1 to 3 (for example, detail level 1 to 3), and converted so that it fits in the display area of the size corresponding to detail levels 1 to 3.

Specifically, the conversion unit 107 generates superimposed information for displaying all messages read from the product information DB 12b in a display area having a size corresponding to the detail level 1, for example, for products with the detail level 1. do. For example, the conversion unit 107 generates superimposition information that can display the entire message in the display area by adjusting the font for display. Further, the conversion unit 107 adjusts the font for products with detail level 2 or 3, summarizes the message read from the product information DB 12b, and displays it in a display area having a size corresponding to detail level 2 or 3. Generate overlay information. For example, the conversion unit 107 extracts appropriate parts from the message read from the product information DB 12b using an extraction-type summary algorithm that skillfully extracts, combines and summarizes important words and sentences from the message read from the product information DB 12b. to generate a summary message. In addition, the conversion unit 107 converts the message into some kind of intermediate expression once, and uses a generative summary model that summarizes the intermediate expression using natural language generation technology, and extracts an appropriate summary message from the message read from the product information DB 12b. may be generated. When the summary message is generated, the conversion unit 107 generates superimposition information for displaying the generated summary message in a display area having a size corresponding to detail level 2 or 3. FIG. Note that the conversion unit 107 may switch the summary algorithm or summary model according to the level of detail, and may change the number of characters to be displayed according to the level of detail.

Further, the conversion unit 107 generates superimposition information for displaying an icon in a display area having a size corresponding to the detail level 4 based on the icon information read from the product information DB 12b for the detail level 4 product. For example, the conversion unit 107 generates superimposition information that can be displayed in the display area by enlarging or reducing an icon based on the icon information. Further, the conversion unit 107 generates superimposition information for displaying a predetermined icon in a display area having a size corresponding to detail level 5 for products with detail level 5 . Note that icon information for displaying a predetermined icon is stored in the information superimposition application 12 a or the storage unit 12 . Through the above processing, the degree-of-association identification unit 103, the distance detection unit 104, the level-of-detail identification unit 105, the first reading unit 106, and the conversion unit 107 perform It functions as an information specifying unit that specifies (generates) superimposed information (display information) regarding a product.

The first superimposition unit 108 superimposes (adds) the superimposition information generated by the conversion unit 107 on the captured image acquired by the image acquisition unit 101 . Note that the first superimposition unit 108 adds superimposition information of each product to a position corresponding to the imaging position of each product detected by the product detection unit 102 in the captured image. The first superimposing unit 108 sends the captured image with the superimposed information to the display unit 14 and displays it on the display unit 14 . FIG. 5B shows a screen example of a captured image to which superimposition information of each product is added. In the screen shown in FIG. 5B, superimposition information is added to products A to C, respectively. In the captured image shown in FIG. 5B, for example, a message (superimposed information) displayed in a large display area is added to product A, and a summary message displayed in a small display area is added to product C. An icon is added to the product B. In addition, since the subject D in the captured image shown in FIG. 5B is a product that cannot be detected by the product detection unit 102, superimposition information is not added to the subject D. FIG.

Note that the information processing apparatus 10 is configured by, for example, a wearable device such as eyeglasses or contact lenses, and the user does not see the surrounding space as an image captured by the camera 13 via the display unit 14, but directly sees the real space. can be configured to see In this case, the first superimposing unit (output unit) 108 sends out only the superimposed information of each product to the display unit 14, and displays it at a position corresponding to the imaging position of each product in the captured image on the display screen of the display unit 14. Let In this case, superimposed information of each product is displayed at a position corresponding to each product in the real space viewed by the user, and information on each product can be provided to the user.
Further, for example, when a message of a data amount corresponding to each detail level is registered in the product information DB 12b as a message for each product, the conversion unit 107 is not required. In this case, the first reading unit 106 reads, from the product information DB 12b, a message corresponding to the detail level specified by the detail level specifying unit 105 for each product detected by the product detecting unit 102, and the first superimposing unit 108 , the read message may be added to the captured image.

In the information processing apparatus 10 of the present embodiment, the screen displayed on the display unit 14 is configured so that superimposition information added to each product in the captured image can be selected via the input unit 15 . The user of the information processing apparatus 10 confirms the superimposed information (message or icon) of each displayed product, and if there is an interesting product or superimposed information, selects the product or superimposed information via the input unit 15. I do. The selection accepting unit 109 accepts selection of a product by the input unit 15 when a selection operation is performed on any product or superimposed information on the screen as shown in FIG. 5B.

The second reading unit 110 reads the display information corresponding to the product for which the selection receiving unit 109 has received the selection from the product information DB 12b. The second reading unit 110 may read only the message of the display information.
The second superimposing unit 111 generates superimposing information for displaying all of the messages read by the second reading unit 110 in a display area having a size corresponding to detail level 1, for example, and displays the generated superimposing information as an image. It superimposes (adds) to the captured image acquired by the acquisition unit 101 . Note that the second superimposing unit 111 adds superimposing information to a position corresponding to the imaging position of the selected product in the captured image. The second superimposing unit 111 sends the captured image with the superimposed information to the display unit 14 and displays it on the display unit 14 . FIG. 5C shows a screen example of a captured image in which product B is selected on the screen of FIG. 5B and superimposed information displaying all messages of product B is added. On the screen shown in FIG. 5C, the display of the superimposed information (message or icon) of products other than the selected product B is terminated and only the superimposed information of product B is displayed, but the configuration is not limited to this. For example, the superimposed information of product B may be superimposed on the superimposed information of products other than product B and displayed. Also, the second superimposing unit 111 may generate superimposing information for displaying the product message in a display area larger than the size corresponding to detail level 1 .

Also here, when the information processing apparatus 10 is configured so that the user can view the real space, the second superimposing unit 111 sends only the superimposing information of the product selected by the user to the display unit 14, and displays the selected product on the display screen. The superimposed information is displayed at a position corresponding to the imaging position of the product. In this case as well, the information about the product is displayed at the position corresponding to the product selected by the user in the real space viewed by the user.

Next, information superimposition processing performed by the information processing apparatus 10 will be described using a flowchart. FIG. 6 is a flowchart showing the procedure of processing performed by the information processing apparatus 10. As shown in FIG. The following processing is executed by the control unit 11 according to a control program including the information superimposition application 12a stored in the storage unit 12 of the information processing device 10. FIG.
A user of the information processing apparatus 10 instructs the activation of the information superimposition application 12a via the input unit 15 . The control unit 11 of the information processing apparatus 10 activates the information superimposition application 12a stored in the storage unit 12 when receiving the instruction to activate the information superimposition application 12a. When the information superimposition application 12a is activated, the control unit 11 activates the camera 13 and acquires an image captured by the camera 13 (S1). The control unit 11 acquires a captured image as shown in FIG. 5A, for example. Note that the camera 13 performs imaging processing at a predetermined timing, and the acquired captured image is sent to the display unit 14 and displayed.

Next, the control unit 11 detects the presence/absence of the product registered in the product information DB 12b in the acquired captured image (S2). The control unit 11 determines whether or not the product registered in the product information DB 12b is included in the captured image (S3). When it is determined that the product is not included (S3: NO), the process proceeds to step S13. , and the acquired captured image is sent to the display unit 14 as it is.

When it is determined that the captured image includes the product registered in the product information DB 12b (S3: YES), the control unit 11 determines whether the user of the information processing device 10 for each product included in the captured image. The degree of association with the attribute is specified from the attribute information DB 12c (S4). Further, the control unit 11 detects the distance from the camera 13 (information processing device 10) at the time of image capturing for each product (S5). Note that the control unit 11 may acquire the captured image in step S1 and detect the distance to each product when detecting the product in the captured image in step S2. Then, the control unit 11 specifies, from the detail level table 12d, the level of detail corresponding to the degree of association specified in step S4 and the distance detected in step S5 for each product (S6).

Next, the control unit 11 reads display information corresponding to each product from the product information DB 12b (S7), and performs conversion processing on the read display information according to the level of detail specified in step S6. to generate superimposition information (S8). Then, the control unit 11 superimposes the generated superimposition information on the captured image acquired in step S1 (S9). As a result, as shown in FIG. 5B, an image in which a message or icon (superimposed information) regarding the product is added to each product in the captured image is displayed on the display unit 14 .

In the information processing apparatus 10 displaying the screen as shown in FIG. 5B, the control unit 11 determines whether or not the input unit 15 has received a selection for the superimposition information of any product (S10). When determining that the selection has not been received (S10: NO), the control unit 11 shifts the processing to step S13. If it is determined that the selection has been received (S10: YES), the control unit 11 reads display information corresponding to the selected product from the product information DB 12b (S11), and displays the read display information (message) in a large display area. Superimposition information for performing is generated and superimposed on the captured image acquired in step S1 (S12). As a result, as shown in FIG. 5C, an image in which a message regarding the selected product is displayed in a large display area is displayed on the display unit 14 .

The control unit 11 determines whether or not to end the process (S13). The case of ending the processing is, for example, the case where the control unit 11 receives an end instruction from the user via the input unit 15 . If it is determined not to end (S13: NO), the control unit 11 returns to the process of step S1, acquires a captured image with the camera 13 (S1), and performs the processes of steps S2 to S12 on the acquired captured image. conduct. When determining to end (S13: YES), the control unit 11 ends the process.

In the present embodiment, it is possible to superimpose and display information (message or icon) related to the product included in the captured image on the image captured by the user using the information processing device 10 . At this time, the information about the product can be displayed in a size and a degree of detail according to the degree of association between the product and the attributes of the user and the distance between the user (information processing device 10) and the product at the time of imaging. Therefore, for example, products with a high degree of relevance to the attributes of the user, or products that are close to the user at the time of imaging may attract a high degree of attention from the user, so information about the product is displayed in detail in a large display area. be able to. In addition, since there is a possibility that the degree of attention of the user is low for products that have a low degree of relevance to the attributes of the user and products that are far away from the user when the image is captured, information about the products can be displayed in a small display area. . In this way, by varying the display mode (display size and degree of detail) of information on products in consideration of the user's attention level, even when a large amount of information is displayed, it does not become difficult to see. In addition, since the display mode of product information can be changed for each user, it is possible to conspicuously display product information that is likely to attract the attention of each user. Therefore, the user can easily find the product that interests him/herself from each product in the captured image.

In this embodiment, when a message regarding a product prepared in advance does not fit in an area to be displayed, the message is summarized and displayed. Therefore, it is possible to efficiently display the information to be notified to the user with a small number of characters, and the user can efficiently obtain information about each product with a message of a small number of characters.

In the present embodiment, the degree of association between each product and the user may be determined by considering, for example, the user's purchase history, hobbies, habits, etc., in addition to attributes including the user's gender and age group. For example, information related to the user's purchase history, hobbies, habits, etc. is acquired by input from the user or acquisition from a website such as a predetermined SNS (Social Network Service). Then, for example, when a product that the user has purchased or a product of the same type (genre) as this product is included in the captured image, a high degree of association with this product may be specified. Further, after specifying the degree of association with the attributes of the user for each product included in the captured image based on the attribute information DB 12c, for products included in the purchase history of the user, the relationship one level higher than the specified degree of association is determined. The degree may be the relevance degree of this product. In this case, it is possible to change the display mode of the information on each product in consideration of personal information such as the user's purchase history, hobbies, and habits.

In this embodiment, the superimposed information added to each product in the captured image may change the display color, font, etc. in addition to changing the size of the display area according to the level of detail. For example, information (superimposed information) of products with a high level of detail (products with level of detail close to 1 in this embodiment) may be displayed in a prominent color or in a prominent font. As a result, it is possible to make the information about the product, which is likely to attract the user's attention, stand out and appeal to the user.

In this embodiment, the information processing apparatus 10 may be configured to use a wearable device such as eyeglasses or contact lenses so that the user can directly see the real space instead of looking through the image captured by the camera 13. . In this case, the control unit 11 displays the superimposed information of each product at a position corresponding to the position of each product in the real space viewed by the user on the display screen of the display unit 14 (for example, eyeglass lenses or contact lenses). The superimposed information of each product is sent to the display unit 14 so as to be displayed. Even with such a configuration, it is possible to provide the user with information about each product in the physical space.

(Embodiment 2)
In the present embodiment, an information processing apparatus will be described that collects and displays superimposed information to be added to each product when an image of the same product being displayed nearby is captured. Since the information processing apparatus of the present embodiment has the same configuration as the information processing apparatus 10 of the first embodiment, the description of the parts common to the first embodiment will be omitted.

FIG. 7 is a block diagram showing functions realized by the control unit 11 of the information processing apparatus 10 of the second embodiment, and FIG. 8 is a schematic diagram showing an example of a display screen of the information processing apparatus 10. As shown in FIG. When the information superimposition application 12a is executed, the control unit 11 of the information processing apparatus 10 of the present embodiment implements the functions of the product classification unit 112 in addition to the functions shown in FIG. Note that in FIG. 7, illustration of each unit other than the product detection unit 102, the degree-of-association identification unit 103, the first reading unit 106, and the product classification unit 112 is omitted. Also in this embodiment, a part of each of these functions may be realized by a dedicated hardware circuit.

The product classification unit 112 classifies the products detected by the product detection unit 102 by genre (type), and identifies a representative product representing each product for each product of the same genre displayed nearby. Specifically, the product classification unit 112 has a genre identification unit 113 , a distance determination unit 114 and a representative product identification unit 115 . The genre identification unit 113 identifies the genre of each product detected by the product detection unit 102 based on the product information DB 12b. Specifically, the genre identification unit 113 reads the genre associated with the product ID of each product detected by the product detection unit 102 from the product information DB 12b. For example, in the case of the captured image shown in FIG. 8A, the product detection unit 102 detects products A, B1 to B4, and C1 to C3 from the captured image. Note that the products B1 to B4 and the products C1 to C3 are products of the same genre, such as products with different colors, products with slightly different shapes, and the like. The genre identification unit 113 identifies the same genre as the genres of the products B1 to B4 detected by the product detection unit 102, and identifies the same genre as the genres of the products C1 to C3.

The distance determining unit (determining unit) 114 determines whether the distance between products (between objects) is less than a predetermined distance for each genre specified by the genre specifying unit 113 . Specifically, the distance determination unit 114 determines whether or not the distance (interval) between each product in the captured image is less than a predetermined distance. Here, "less than a predetermined distance" may be, for example, a distance at which it is possible to determine whether the products are displayed side by side as the same product or displayed at a separate position. In the case of the captured image shown in FIG. 8A, for example, the distance determination unit 114 determines that the distance between the products B1 and B4 and the distance between the products C1 and C2 are less than a predetermined distance. It is determined that the distance is greater than or equal to the predetermined distance.

The representative product identification unit 115 identifies one product (representative product) representing a plurality of products determined by the distance determination unit 114 that the distance between the products is less than the predetermined distance. The representative product identification unit 115 identifies, as representative products, for example, a product closest to the camera 13, a product photographed in the largest size, a product photographed from the front, and the like. In the case of the captured image shown in FIG. 8A, for example, the representative product identification unit 115 identifies product B1 as a representative product of products B1 to B4 and product C1 as a representative product of products C1 to C2. As for a product that is separated from other products of the same genre by a predetermined distance or more, the representative product identification unit 115 identifies this product as a representative product.

Then, the degree-of-association specifying unit 103, the distance detecting unit 104, the level-of-detail specifying unit 105, the first reading unit 106, the converting unit 107, and the first superimposing unit 108 of this embodiment are the representative product specifying unit 115 of the product classifying unit 112. Each process is performed for each product (representative product) specified by . As a result, as shown in FIG. 8B, with respect to the captured image, for each of the product B1, which is the representative product of the product A and the products B1 to B4, and the product C1, which is the representative product of the products C1 to C2, and the product C3, Superimposition information can be added. Note that the product classification unit 112 may be configured to display one piece of superimposed information by aggregating superimposed information to be displayed on each product for a plurality of identical products displayed nearby.

Next, information superimposition processing performed by the information processing apparatus 10 of this embodiment will be described. FIG. 9 is a flow chart showing part of the procedure of processing performed by the information processing apparatus 10 . The process shown in FIG. 9 is obtained by adding the processes of steps S21 to S24 between steps S3 and S4 in the process of the first embodiment shown in FIG. In FIG. 9, illustration of steps S1 to S2 and S5 to S13 in FIG. 6 is omitted.
When the control unit 11 of the information processing apparatus 10 of the present embodiment determines that the captured image includes the product registered in the product information DB 12b in step S3 (S3: YES), the The genre of each product is specified for each product (S21). For example, the control unit 11 reads the genre of each product from the product information DB 12b. Next, the control unit 11 measures the distance between each product for each genre specified in step S21 (S22), and determines whether the measured distance is less than a predetermined distance (S23).

When it is determined that there is a product whose distance between products is less than the predetermined distance (S23: YES), the control unit 11 specifies one representative product from among the products (S24). Then, the control unit 11 performs the processing after step S4 on the representative product specified in step S24. If it is determined that the distance between each product is not less than the predetermined distance (S23: NO), the control unit 11 skips the processing of step S24 and performs the processing after step S4 for each product included in the captured image. conduct.

In this embodiment, the same effects as those of the first embodiment described above can be obtained. In addition, in the present embodiment, product information (messages or icons) can be collectively displayed for products of the same genre that are within a predetermined range (the distance between products is less than the predetermined distance). Therefore, by collectively displaying the superimposed information to be added to each product, each superimposed information can be displayed in a more legible state. Therefore, when the user searches for a product that interests him/herself in the captured image, the user can find the product more easily.

(Embodiment 3)
In this embodiment, when adding the same superimposed information (message or icon) to each product in a captured image, an information processing apparatus that aggregates and displays the superimposed information will be described. Since the information processing apparatus of the present embodiment has the same configuration as the information processing apparatus 10 of the first embodiment, the description of the parts common to the first embodiment will be omitted.

FIG. 10 is a block diagram showing functions realized by the control unit 11 of the information processing apparatus 10 of the third embodiment. When the information superimposition application 12a is executed, the control unit 11 of the information processing apparatus 10 of the present embodiment implements the functions of the information aggregation unit 116 in addition to the functions shown in FIG. Note that FIG. 10 omits illustration of each unit other than the conversion unit 107, the first superimposition unit 108, and the information aggregation unit 116. FIG. Also in this embodiment, a part of each of these functions may be realized by a dedicated hardware circuit.

The information aggregating unit 116 performs a process of aggregating the superimposed information generated by the conversion unit 107 for each product that appears close in the captured image into one when the superimposed information is duplicated. Specifically, the information aggregating unit 116 has an identity determination unit 117 , a distance determination unit 118 and a representative product identification unit 119 . The identity determination unit 117 determines whether the superimposition information generated by the conversion unit 107 for each product detected by the product detection unit 102 is the same. The identity determination unit 117 determines whether or not the display contents of the message, icon, etc. are the same in the superimposed information of each product.

A distance judgment unit (judgment unit) 118 judges whether or not the distance between products determined by the identity determination unit 117 to have the same superimposition information is less than a predetermined distance. Also here, the distance determination unit 118 determines whether or not the distance (interval) between the products in the captured image is less than the predetermined distance. The representative product identification unit 119 identifies one product (representative product) representing a plurality of products determined by the distance determination unit 118 that the distance between the products is less than the predetermined distance. Then, in the captured image, the first superimposing unit 108 of the present embodiment adds the conversion unit 107 only to the position corresponding to the photographing position of each product (representative product) specified by the representative product specifying unit 119 of the information aggregating unit 116 . Adds superimposed information generated by Therefore, also in the present embodiment, a captured image as shown in FIG. 8B can be displayed as in the above-described second embodiment.

For example, when the products B1 to B4 in FIG. 8A are the same products, the conversion unit 107 of this embodiment generates the same superimposition information for the products B1 to B4. Then, the information aggregation unit 116 aggregates the same superimposition information for the products B1 to B4 into one, and the first superimposition unit 108 adds the superimposition information only to the product B1, which is the representative product of the products B1 to B4, for example. . As a result, as shown in FIG. 8B, the superimposed information of the products B1 to B4 is aggregated and added only to the representative product B1, and the superimposed information of the products C1 to C2 is aggregated and added only to the representative product C1. Therefore, in the present embodiment, even if the products are different or belong to different genres, if the superimposed information is the same and the distance between the products is less than a predetermined distance, the superimposed information is displayed only on the representative products ( addition) can be done. In addition to being displayed at a position corresponding to one representative product, the aggregated superimposition information may be displayed at a position where multiple products to which the same superimposition information should be added can be included.

Next, information superimposition processing performed by the information processing apparatus 10 of this embodiment will be described. FIG. 11 is a flow chart showing part of the procedure of processing performed by the information processing apparatus 10 . The process shown in FIG. 11 is obtained by adding the processes of steps S31 to S34 between steps S8 and S9 in the process of the first embodiment shown in FIG. In FIG. 11, illustration of steps S1 to S7 and S10 to S13 in FIG. 6 is omitted.

After the process of step S8, the control unit 11 of the information processing apparatus 10 of the present embodiment determines whether or not there is the same superimposition information in the superimposition information of each product in the captured image generated in step S8 (S31 ). If it is determined that there is the same superimposition information (S31: YES), the control unit 11 measures the distance between each product with the same superimposition information (S32), and if the measured distance is less than the predetermined distance. It is determined whether or not there is (S33). If it is determined that there is a product whose distance between products is less than the predetermined distance (S33: YES), the control unit 11 identifies one representative product from among these products (S34). Then, the control unit 11 superimposes the superimposition information generated in step S8 on the identified representative product at the position corresponding to the representative product in the captured image (S9). As a result, a screen as shown in FIG. 8B is displayed on the display unit 14, and the control unit 11 performs the processes after step S10. On the other hand, if it is determined that there is no identical superimposed information (S31: NO), or if it is determined that the distance between each product is not less than the predetermined distance (S33: NO), the control unit 11 shifts the processing to step S9. , the superimposition information generated in step S8 for each product is superimposed on the position corresponding to each product in the captured image (S9). Then, the control unit 11 performs the processes after step S10.

In this embodiment, the same effects as those of the first embodiment described above can be obtained. Further, in the present embodiment, when the information (message or icon) to be added to products within a predetermined range (distance between products is less than a predetermined distance) is the same, one piece of information can be displayed collectively. can. That is, in the present embodiment, even if the products are different products or products of different genres, if each product is within a predetermined range and the superimposed information is the same, the superimposed information is added only to the representative product, and unnecessary superimposed information is removed. display can be avoided. Therefore, the same superimposed information is suppressed from being displayed multiple times, so each superimposed information can be displayed in a more legible state. As a result, the user can more easily find the product of interest to the user in the captured image.

(Embodiment 4)
In this embodiment, an information processing apparatus that translates into a language used by a user when generating superimposed information to be added to each product in a captured image will be described. Since the information processing apparatus of the present embodiment has the same configuration as the information processing apparatus 10 of the first embodiment, the description of the parts common to the first embodiment will be omitted.

FIG. 12 is a block diagram showing functions realized by the control unit 11 of the information processing apparatus 10 of the fourth embodiment, and FIG. 13 is a schematic diagram showing an example of a display screen of the information processing apparatus 10. As shown in FIG. When the information superimposition application 12a is executed, the control unit 11 of the information processing apparatus 10 of this embodiment implements the functions of the language acquisition unit 120 in addition to the functions shown in FIG. It should be noted that FIG. 12 shows units other than the level-of-detail specifying unit 105, the first reading unit 106, the converting unit 107, the first superimposing unit 108, the second reading unit 110, the second superimposing unit 111, and the language acquisition unit 120. omitted. Also in this embodiment, a part of each of these functions may be realized by a dedicated hardware circuit.

The language acquiring unit (language specifying unit) 120 acquires (specifies) the type of language (for example, Japanese, English, Chinese, etc.) used by the user of the information processing device 10 . The language acquisition unit 120 acquires the language used by the user, for example, through input by the user or acquisition from an OS installed in the information processing apparatus 10 or an application executable in the information processing apparatus 10 . A language to be used is set in the OS or the application program, and the language acquisition unit 120 acquires the language to be used from the OS or the application program. The conversion unit 107 of the present embodiment converts the display information read by the first reading unit 106 for each product detected by the product detection unit 102 into a display size corresponding to the level of detail specified by the level-of-detail specifying unit 105 . Convert to fit in area. Then, the conversion unit (language conversion unit) 107 translates the converted display information into the language acquired by the language acquisition unit 120 to generate superimposition information. After that, the first superimposition unit 108 adds the generated superimposition information to the captured image acquired by the image acquisition unit 101 and outputs the captured image to the display unit 14 , and displays the captured image to which the superimposition information is added on the display unit 14 . . As a result, for example, when the language used by the user is English, an image in which English superimposed information is added to each product in the photographed image is displayed on the display unit 14 as shown in FIG. 13 . Note that the conversion unit 107 of the present embodiment first translates the display information read by the first reading unit 106 into the language acquired by the language acquisition unit 120, and converts the display information after translation according to the specified detail level. The configuration may be such that the conversion is performed so as to fit within the display area of the size.

Also, the second superimposing unit 111 of the present embodiment translates the message read by the second reading unit 110 into the language acquired by the language acquisition unit 120 . Then, the second superimposition unit 111 generates superimposition information for displaying the translated message in a display area having a size corresponding to detail level 1, for example. After that, the second superimposing unit 111 adds the generated superimposing information to the captured image acquired by the image acquiring unit 101 and outputs the captured image to the display unit 14 , and displays the captured image to which the superimposing information is added on the display unit 14 . . As a result, an image in which a message about the product is displayed (added) in the language used by the user is displayed on the product selected by the user in the captured image.

The control unit 11 of the information processing apparatus 10 of this embodiment performs the same processing as the processing shown in FIG. Note that, in step S8, the control unit 11 of the present embodiment adjusts the display information read out from the product information DB 12b in step S7 for each product detected in step S2 according to the level of detail specified in step S6. Superimposition information is generated by conversion processing and translation into the language used by the user. Note that the language used by the user may be set or obtained in advance, or may be obtained when the superimposition information is generated. As a result, as shown in FIG. 13, an image in which an English message or icon (superimposed information) regarding the product is added to each product in the captured image is displayed on the display unit 14 . Further, in step S12, the control unit 11 of the present embodiment translates the display information (message) read out from the product information DB 12b in step S11 into the language used by the user and displays it in a large display area. Transformation processing is performed to generate superimposition information. As a result, an image in which a message regarding product B is displayed in the language used by the user on the screen shown in FIG. 5C, for example, is displayed on the display unit 14 .

In this embodiment, the same effects as those of the first embodiment described above can be obtained. In addition, in the present embodiment, the information (superimposed information) of each product to be added to the product in the captured image is translated into the language used by the user and displayed. The configuration of this embodiment can also be applied to the second and third embodiments, and even when applied to the second and third embodiments, the same effects as those of the second and third embodiments can be obtained.

(Embodiment 5)
In the present embodiment, when detecting whether or not a product registered in the product information DB 12b is included in the captured image, the display area of the display unit 14 is subjected to product detection processing (detection range). ) will be described. The information processing apparatus of this embodiment has the same configuration as the information processing apparatus 10 of the first embodiment. Further, when the information superimposition application 12a is executed, the control unit 11 of the information processing apparatus 10 of the present embodiment realizes each function shown in FIG. 4 and performs the same processing as each function in the first embodiment. The description of the parts common to the first embodiment is omitted.

FIG. 14 is an explanatory diagram of processing performed by the information processing apparatus 10 of the fifth embodiment. In the information processing apparatus 10 of the present embodiment, in the display area of the display unit 14, the processing target area (detection area) in which the product detection unit 102 detects the presence or absence of products registered in the product information DB 12b is limited. In FIG. 14, the detection area 14a is indicated by a dashed line, and in this embodiment, only the subject appearing within the detection area 14a is processed. Therefore, since the subject appearing at the edge of the captured image can be excluded from the processing target, only the subject (merchandise) that is likely to attract the attention of the user can be the processing target. There is a high possibility that the user does not pay attention to products that appear in places other than the detection area 14a, such as the edges of the captured image. Do not superimpose on Therefore, it is possible to conspicuously display the product information that the user is likely to pay attention to, and to reduce the processing load of the information processing apparatus 10 by not performing unnecessary processing.

Therefore, the product detection unit 102 of this embodiment detects whether or not the product registered in the product information DB 12b is included in the set detection area in the captured image acquired by the image acquisition unit 101 . Note that the control unit 11 of the present embodiment sets the range of the detection area when the information superimposition application 12a is activated or when the information superimposition application 12a is installed. Also, the range of the detection area may be set by, for example, user input received by the input unit (reception unit) 15, or a preset range may be set.
The configuration of this embodiment can also be applied to Embodiments 2 to 4, and even when applied to Embodiments 2 to 4, the same effects as those of Embodiments 2 to 4 can be obtained.

(Embodiment 6)
In this embodiment, when any product (or superimposed information added to the product) in the captured image is selected via the captured image to which the superimposed information is added, the selected product is selected. An information processing apparatus that stores the user's attributes in association with each other will be described. Since the information processing apparatus of the present embodiment has the same configuration as the information processing apparatus 10 of the first embodiment, the description of the parts common to the first embodiment will be omitted.

FIG. 15 is a block diagram showing functions realized by the control unit 11 of the information processing apparatus 10 of the sixth embodiment. When the information superimposition application 12a is executed, the control unit 11 of the information processing apparatus 10 of this embodiment implements the functions of the association unit 121 in addition to the functions shown in FIG. Note that in FIG. 15, illustration of each unit other than the selection receiving unit 109, the second reading unit 110, the second superimposing unit 111, and the associating unit 121 is omitted. Also in this embodiment, a part of each of these functions may be realized by a dedicated hardware circuit.

The selection reception unit (selection information acquisition unit) 109 of the present embodiment performs a selection operation on any product or superimposition information of the product via a display screen on which a captured image in which superimposition information is added to each product is displayed. is received by the input unit 15 . When the input unit 15 receives a selection operation, the selection reception unit 109 acquires selection information indicating the selected product. When the selection receiving unit 109 receives the selection information, the associating unit 121 associates the selected product with attributes of the user of the information processing device 10, and stores them in a predetermined database ( (not shown). Specifically, the associating unit 121 reads the product ID of the selected product from the product information DB 12b, reads the user attribute from the information superimposition application 12a or the storage unit 12, and associates the read product ID and the user attribute. and register it in the database.

The control unit 11 of the information processing apparatus 10 of this embodiment performs the same processing as the processing shown in FIG. Note that when the control unit 11 of the present embodiment determines in step S10 that the selection for the superimposition information of any product has been received on the screen being displayed (S10: YES), the product ID of the selected product and the user are registered in a predetermined database in association with the attributes of After that, the control unit 11 performs the processing after step S11. As a result, when the user selects an item of interest in the captured image, the selected item can be stored in the database. Therefore, since the selection history of the product selected by the user can be saved, the degree of association between each product and the user's attribute can be reset to a value according to the user using the selection history. Also, by registering the selected product in association with the attribute of the user, based on this association, the degree of association between each product registered in the attribute information DB 12c and the user attribute can be set again. can be done. Therefore, in the information superimposition processing that follows, the degree of association between each product and each user becomes a more appropriate value, so the product to which superimposition information should be added can be more appropriately specified according to the user.

In this embodiment, the same effects as those of the first embodiment described above can be obtained. In addition, in the present embodiment, the degree of association between each product and user attributes can be set using the product selection history of the user, so information on products to be provided to the user can be provided appropriately. Therefore, the user can more easily find the product of interest to the user in the captured image. The configuration of the present embodiment can also be applied to Embodiments 2 to 5, and even when applied to Embodiments 2 to 5, the same effects as those of Embodiments 2 to 5 can be obtained.

(Embodiment 7)
In this embodiment, a system in which the processing performed by the information processing apparatus 10 of the first embodiment described above is performed by both the information processing apparatus 10 and the server apparatus 20 will be described. Since the information processing apparatus of the present embodiment has the same configuration as the information processing apparatus 10 of the first embodiment, the description of the parts common to the first embodiment will be omitted.

FIG. 16 is a block diagram showing a configuration example of the information processing device 10 and the server device 20 according to the seventh embodiment. In the system of this embodiment, the information processing device 10 and the server device 20 can be connected via a network N such as the Internet. The information processing apparatus 10 of this embodiment is the same as that of the first embodiment except that the storage unit 12 stores only the information superimposed application 12a.

The server device 20 is, for example, a server computer, a personal computer, or the like, and includes a control section 21, a storage section 22, a communication section 23, etc. These sections are interconnected via a bus. The control unit 21 includes one or more processors such as a CPU, MPU, or GPU, and appropriately executes control programs stored in the storage unit 22 to perform various information processing and control processes to be performed by the server device 20. etc. The storage unit 22 includes RAM, flash memory, hard disk, SSD, and the like. The storage unit 22 stores in advance control programs executed by the control unit 21 and various data necessary for executing the control programs. The storage unit 22 also temporarily stores data and the like generated when the control unit 21 executes the control program.

The control programs stored in the storage unit 22 include an information generation program 22a. Data stored in the storage unit 22 includes a product information DB 22b, an attribute information DB 22c, and a detail level table 22d having the same configuration as the product information DB 12b, attribute information DB 12c, and detail level table 12d of the first embodiment. The information generation program 22a, the product information DB 22b, the attribute information DB 22c, and the detail level table 22d are acquired from an external device via the communication section 23 and stored in the storage section 22, for example. In addition, when the server device 20 includes a reading unit or the like for reading information stored in a portable storage medium, the information generation program 22a, the product information DB 22b, the attribute information DB 22c, and the detailed level table 22d are read from the portable storage medium. may be stored in the storage unit 22. The product information DB 22b, the attribute information DB 22c, and the detail level table 22d may be stored in an external storage device or the like connected to the server device 20, or may be stored in another storage device with which the server device 20 can communicate via the network N. may be stored on the device.

The communication unit 23 is an interface for connecting to the network N, and transmits and receives information to and from an external device via the network N.
In addition to the configuration described above, the server device 20 may include a display unit such as a liquid crystal display or an organic EL display, and an input unit such as a keyboard or mouse.

In the present embodiment, the information processing device 10 performs imaging processing using the camera 13, transmits the acquired captured image to the server device 20 via the network N, and the server device 20, based on the received captured image, The information superimposition processing performed by the information processing apparatus 10 of No. 1 is performed. Specifically, the server device 20 generates superimposed information to be added to the captured image by the information superimposing process, and transmits the superimposed information to the information processing device 10. The information processing device 10 superimposes the received superimposed information on the captured image. displayed. That is, the server device 20 of the present embodiment generates superimposition information to be superimposed on the captured image based on the captured image acquired from the information processing device 10 and transmits the generated superimposition information to the information processing device 10 . Further, when the information processing apparatus 10 receives superimposition information from the server device 20 while displaying the captured image captured by the camera 13 on the display unit 14, the information processing apparatus 10 superimposes the received superimposition information on the captured image being displayed. indicate.

In the present embodiment, the control unit 11 of the information processing apparatus 10 executes the information superimposition application 12a to perform the image acquisition unit 101, the first superimposition unit 108, the selection reception unit 109, and the second superimposition unit 111 in FIG. Realize each function. 4, the control unit 21 of the server device 20 executes the information generation program 22a to perform , the functions of the conversion unit 107 and the second reading unit 110 are realized. Also in this embodiment, a part of each of these functions may be realized by a dedicated hardware circuit.

In the information processing apparatus 10 of this embodiment, the control unit 11 transmits the acquired captured image to the server device 20 via the network N when the image acquisition unit 101 acquires the captured image. In server device 20 , product detection unit 102 , degree-of-relevance identification unit 103 , distance detection unit 104 , level-of-detail identification unit 105 , first reading unit 106 , and conversion unit 107 implemented by control unit 21 receive data from information processing device 10 . Each processing is performed on the picked-up image. Note that the distance detection unit 104 in the server device 20 may detect the distance to each product based on the captured image. For example, the distance detection unit 104 detects the distance to each product based on the size of the imaging area of each product (subject) in the captured image. Further, in the case where the information processing device 10 is configured to detect the distance to each product at the time of imaging by a distance sensor or the like, the control unit 11 of the information processing device 10 transmits the distance to each product together with the captured image to the server device 20. do. In this case, the distance detection unit 104 in the server device 20 can receive the distance to each product in the captured image from the information processing device 10 .

Further, when the conversion unit 107 generates superimposition information, the control unit 21 of the server device 20 transmits the generated superimposition information to the information processing device 10 via the network N. FIG. Then, in the information processing apparatus 10 , the first superimposing unit 108 realized by the control unit 11 superimposes (adds) superimposition information received from the server device 20 on the captured image acquired by the image acquisition unit 101 and displays it on the display unit 14 . send to Accordingly, also in the information processing apparatus 10 of the present embodiment, as shown in FIG. 5B, it is possible to display a captured image in which superimposed information (message or icon) is added to each product. Note that the superimposition information transmitted by the server device 20 to the information processing device 10 includes information to be superimposed and position information on the screen to be superimposed. , appropriate information can be superimposed on an appropriate position on the captured image being displayed.

Further, in the information processing apparatus 10 of the present embodiment, the control unit 11 notifies the server device 20 via the network N of the product for which the selection reception unit 109 has received selection. For example, the control unit 11 transmits the product ID of the selected product to the server device 20 . In the server device 20 , the second reading section 110 implemented by the control section 21 performs processing based on the product ID received from the information processing device 10 . Specifically, the second reading unit 110 reads the display information (message) corresponding to the product with the product ID received from the information processing device 10 from the product information DB 22b. Note that the second reading unit 110 in the server device 20 generates superimposition information for displaying all of the read messages in a display area having a size corresponding to the detail level 1, for example, and sends the generated superimposition information to the network N It transmits to the information processing apparatus 10 via. Then, in the information processing apparatus 10 , the second superimposing unit 111 realized by the control unit 11 superimposes (adds) the superimposition information received from the server device 20 on the captured image acquired by the image acquisition unit 101 and displays it on the display unit 14 . send to Thus, also in the information processing apparatus 10 of the present embodiment, superimposition information (message) can be added to the product selected by the user in the captured image being displayed, as shown in FIG. 5C.

Next, information superimposition processing performed by the information processing device 10 and the server device 20 in the system of the present embodiment will be described based on a flowchart. 17 and 18 are flowcharts showing procedures of processing performed by the information processing device 10 and the server device 20. FIG. 17 and 18, the left side shows the processing performed by the information processing apparatus 10, and the right side shows the processing performed by the server device 20. FIG.

When the information superimposition application 12a stored in the storage unit 12 is activated, the control unit 11 of the information processing apparatus 10 activates the camera 13, acquires a captured image with the camera 13 (S41), and displays the acquired captured image. is transmitted to the server device 20 via the network N (S42). The control unit 21 of the server device 20 receives the captured image transmitted by the information processing device 10 (S43), and performs the same processing as steps S2 to S8 in FIG. 6 described in the first embodiment on the received image. to generate superimposition information to be added to the captured image (S44-S50). Note that if the control unit 21 determines in step S45 that the product registered in the product information DB 22b is not included in the received captured image (S45: NO), the control unit 21 does nothing and performs step S43. Return to processing.

The control unit 21 of the server device 20 transmits the generated superimposition information to the information processing device 10 via the network N (S51). The control unit 11 of the information processing device 10 receives the superimposition information transmitted by the server device 20 (S52), and superimposes the received superimposition information on the captured image being displayed on the display unit 14 (S53). As a result, as shown in FIG. 5B, each product in the captured image can be displayed with a message or icon (superimposed information) relating to the product added.

Then, the control unit 11 of the information processing device 10 determines whether or not selection of any product or superimposed information of the product has been accepted on the screen being displayed (S54). S54: NO), the process proceeds to step S62. On the other hand, if it is determined that the selection has been accepted (S54: YES), the control unit 11 transmits, for example, the product ID of the selected product to the server device 20 (S55). The control unit 21 of the server device 20 receives the product ID transmitted by the information processing device 10 (S56), and reads display information corresponding to the product with the received product ID from the product information DB 22b (S57). Then, the control unit 21 generates superimposition information for displaying the read display information (message) in a large display area (S58), and transmits the generated superimposition information to the information processing apparatus 10 via the network N (S59). .

The control unit 11 of the information processing device 10 receives the superimposition information transmitted by the server device 20 (S60), and superimposes the received superimposition information on the captured image being displayed on the display unit 14 (S61). As a result, as shown in FIG. 5C, the message regarding the product selected by the user can be displayed in a large display area in the captured image being displayed. The control unit 11 of the information processing apparatus 10 determines whether or not to end the process (S62), and if it determines not to end the process (S62: NO), returns to the process of step S41, and if it determines to end the process (S62: YES), terminate the process.

In this embodiment, the same effects as those of the first embodiment described above can be obtained. That is, among the products included in the image captured by the user using the information processing device 10, the product that has a higher degree of relevance to the user and is closer to the user at the time of image capturing is displayed in a larger display area with information (message) related to the product. or icon) can be superimposed and displayed. Therefore, the information of the product that the user is likely to pay attention to can be conspicuously displayed, so that the user can easily find out the product that the user is interested in from the products in the captured image.

In the present embodiment, the captured image transmitted from the information processing device 10 to the server device 20 may be the captured image acquired by the camera 13 or the feature amount data extracted from the captured image. The feature amount data is data for detecting a product (subject) in an image in the server device 20 . In this case, the control unit 11 of the information processing device 10 performs processing for extracting feature amount data from the captured image acquired by the camera 13 and transmits the generated feature amount data to the server device 20 . Then, the server device 20 detects whether or not there is a product registered in the product information DB 12b for the received feature amount data, and performs processing for generating superimposition information for the detected product.

In the present embodiment, the processing performed by the information processing device 10 and the server device 20 are not limited to the configurations described above. For example, the information processing device 10 acquires an image captured by the camera 13, performs processing for detecting whether or not a product is included in the acquired image, and sends information (for example, product ID) related to the detected product to the server device. 20 may be sent. In this case, the server device 20 generates superimposed information about the product detected by the information processing device 10 and transmits the generated superimposed information to the information processing device 10 . Then, the information processing device 10 receives the superimposition information generated by the server device 20, adds it to the captured image, and displays it.

In addition, the information processing apparatus 10 acquires a captured image with the camera 13, detects whether or not a product is included in the acquired captured image, identifies the detail level of each detected product, and identifies each identified product. The product detail level may be transmitted to the server device 20 . In this case, the server device 20 generates superimposition information for each product based on the detail level of each product specified by the information processing device 10 and transmits the generated superimposition information to the information processing device 10 . Then, the information processing device 10 receives the superimposition information generated by the server device 20, adds it to the captured image, and displays it. Note that whether the product information DB, the attribute information DB, and the detail level table should be stored in the storage unit 12 of the information processing device 10 or the storage unit 22 of the server device 20 depends on the processing performed by the information processing device 10 and the server device 20, respectively. You can change whether to store in Also, which process should be performed by the information processing device 10 and the server device 20 can be appropriately changed by an administrator who manages the product information DB, the attribute information DB, and the detail level table.

The system of this embodiment may be configured with a digital signage having a display unit and a camera connected to the digital signage. In this case, the camera acquires the captured image and transmits it to the digital signage, and the digital signage performs information superimposition processing on the received captured image, adds the superimposed information to each product in the captured image, and displays it on the display unit. indicate. Such a configuration can also provide the user with screens as shown in FIGS. 5B and 5C.

The configuration of the present embodiment can also be applied to Embodiments 2 to 6, and even when applied to Embodiments 2 to 6, the same effects as those of Embodiments 2 to 6 can be obtained. When applied to the second to sixth embodiments, the processing to be performed by the information processing device 10 and the server device 20 can be appropriately changed according to each embodiment.

In each of the above-described embodiments, when detecting each product in the captured image, characters (text data) included in the captured image may be detected, and each product may be identified by the detected characters. For example, when a pop-up advertisement is attached to each product, it is possible to detect each product or the type of each product by recognizing the characters described in this pop-up advertisement. Further, for example, an image of an outdoor scene is captured, the store signboard etc. in the obtained captured image are recognized (specified), and information about each store is superimposed on the captured image according to the recognized signboard etc. can do. In this case, it is possible to specify the name of the store by detecting the characters written on the signboard, and to provide the user with information about the specified store.

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the meaning described above, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

REFERENCE SIGNS LIST 10 information processing device 11 control unit 12 storage unit 13 camera 15 input unit (reception unit)
101 image acquisition unit 102 product detection unit (detection unit)
106 first reading unit (information acquisition unit)
107 converter (language converter)
108 first superimposition unit (output unit)
109 selection reception unit (selection information acquisition unit)
114 distance judgment unit (judgment unit)
118 distance judgment unit (judgment unit)
120 language acquisition unit 121 association unit

Claims (14)

an image acquisition unit that acquires an image;
a detection unit that detects an object present in the acquired image;
an information specifying unit that specifies display information of the object according to the detected distance to the object and attributes of the user;
an output unit that outputs the identified display information ;
a determination unit that, when the detection unit detects a plurality of objects of the same or the same genre, determines whether or not the distance between the plurality of objects is less than a predetermined distance;
with
The information specifying unit specifies the display information according to the distance to any one of the plurality of objects and the attribute of the user when it is determined that the distance between the plurality of objects is less than a predetermined distance. information processing equipment. an image acquisition unit that acquires an image;
a detection unit that detects an object present in the acquired image;
an information specifying unit that specifies display information of the object according to the detected distance to the object and attributes of the user;
an output unit that outputs the identified display information;
When the information specifying unit specifies the display information for each of a plurality of objects, the plurality of the display information are the same, and the distance between the objects for which the display information is specified is less than a predetermined distance. a judgment unit for judging whether or not
with
When it is determined that the plurality of display information are the same and the distance is less than a predetermined distance, the output unit outputs one display information at a position corresponding to the plurality of objects in the image. An information processing device that outputs an instruction to display . a selection information acquisition unit for acquiring selection information indicating whether or not the display information output by the output unit is selected;
an associating unit that associates the acquired selection information with the attributes of the user;
The information processing apparatus according to claim 1 or 2, comprising: an image acquisition unit that acquires an image;
a detection unit that detects an object present in the acquired image;
an information specifying unit that specifies display information of the object according to the detected distance to the object and attributes of the user;
an output unit that outputs the identified display information;
a selection information acquisition unit for acquiring selection information indicating whether or not the display information output by the output unit is selected;
an associating unit that associates the acquired selection information with the attributes of the user;
Information processing device. The information specifying unit
an information acquisition unit that acquires display information of the detected object;
5. Any one of claims 1 to 4, further comprising a conversion unit that converts the display information acquired by the information acquisition unit so as to fit in a display area having a size according to the distance to the object and the attributes of the user. 1. The information processing device according to one . The information processing apparatus according to any one of claims 1 to 5 , wherein the output unit outputs an instruction to display the display information at a position corresponding to the object in the image. a language identification unit that identifies the language used by the user;
The information according to any one of claims 1 to 6 , further comprising: a language conversion unit that converts the display information specified by the information specifying unit into display information according to the language specified by the language specifying unit. processing equipment. The information processing apparatus according to any one of claims 1 to 7 , further comprising: a reception unit that receives a detection range in which the detection unit detects an object in the image acquired by the image acquisition unit. to the computer,
get the image,
detecting an object present in the acquired image;
specifying display information of the object according to the detected distance to the object and user attributes;
A process of outputting the identified display information,
when multiple objects of the same or the same genre are detected in the image, determining whether the distance between the multiple objects is less than a predetermined distance;
If it is determined that the distance between the plurality of objects is less than a predetermined distance, the display information is specified according to the distance to any one of the plurality of objects and the attribute of the user.
A program that causes an action to take place. to the computer,
get the image,
detecting an object present in the acquired image;
specifying display information of the object according to the detected distance to the object and user attributes;
A process of outputting the identified display information,
When the display information for each of a plurality of objects is specified, it is determined whether or not the plurality of the display information are the same and the distance between the objects for which the display information is specified is less than a predetermined distance. death,
When it is determined that the plurality of pieces of display information are the same and the distance is less than a predetermined distance, an instruction is output to display one piece of the display information at a position corresponding to the plurality of objects in the image. do
A program that causes an action to take place. to the computer,
get the image,
detecting an object present in the acquired image;
specifying display information of the object according to the detected distance to the object and user attributes;
outputting the identified display information;
Acquire selection information indicating whether or not the output display information is selected,
Correlate the acquired selection information and the attributes of the user
A program that causes an action to take place. an imaging unit;
a detection unit that detects an object present in an image obtained by imaging;
an information specifying unit that specifies display information of the object according to the detected distance to the object and attributes of the user;
an output unit that outputs the identified display information ;
a determination unit that, when the detection unit detects a plurality of objects of the same or the same genre, determines whether or not the distance between the plurality of objects is less than a predetermined distance;
with
The information specifying unit specifies the display information according to the distance to any one of the plurality of objects and the attribute of the user when it is determined that the distance between the plurality of objects is less than a predetermined distance. information processing system. an imaging unit;
a detection unit that detects an object present in an image obtained by imaging;
an information specifying unit that specifies display information of the object according to the detected distance to the object and attributes of the user;
an output unit that outputs the identified display information;
When the information specifying unit specifies the display information for each of a plurality of objects, the plurality of the display information are the same, and the distance between the objects for which the display information is specified is less than a predetermined distance. a judgment unit for judging whether or not
with
When it is determined that the plurality of display information are the same and the distance is less than a predetermined distance, the output unit outputs one display information at a position corresponding to the plurality of objects in the image. An information processing system that outputs instructions to display an imaging unit;
a detection unit that detects an object present in an image obtained by imaging;
an information specifying unit that specifies display information of the object according to the detected distance to the object and attributes of the user;
an output unit that outputs the identified display information;
a selection information acquisition unit for acquiring selection information indicating whether or not the display information output by the output unit is selected;
an associating unit that associates the acquired selection information with the attributes of the user;
An information processing system comprising

Images