JP7759038B2 - Information Processing Systems - Google Patents

Description

The present invention relates to an information processing system that does not reduce recognition accuracy when identifying objects from image information.

In convenience stores, supermarkets, and other stores, it is common for products and other objects to be sold by placing them on display shelves. This display method involves displaying multiple objects horizontally to make them more visible to customers, or vertically so that even if one object is purchased, another person can purchase the same object. Managing where and how many objects are displayed on the display shelves is important in terms of sales strategies for the objects.

Therefore, in order to understand the actual display status of objects in a store, one method is to photograph the display shelves with a camera and automatically identify the displayed objects from the captured image information. For example, one method uses image recognition technology on images of the store's display shelves based on specimen images of each object. Examples of such prior art include Patent Documents 1 and 2 listed below.

Japanese Patent Application Publication No. 5-342230 Japanese Patent Application Publication No. 5-334409

The invention of Patent Document 1 is a system that assists even those without knowledge in determining on which display shelf an object should be displayed. As such, it is possible to determine where an object should be displayed, but it does not identify the object that is displayed. Patent Document 2 is a system that supports the input of object images in a shelf allocation support system that supports the display of objects. However, the system of Patent Document 2 only supports the input of object images when using the shelf allocation support system, and even if this system is used, it is not possible to grasp the specific display status of objects.

In addition to Patent Documents 1 and 2, there is also technology that uses image recognition processing technology to identify displayed objects from image information captured on display shelves. This is beneficial in that it allows for understanding the actual display situation in a store.

In conventional technology, when identifying an object using image recognition processing technology, a rectangular area where the object is likely to be located is detected from image information captured of a display shelf, and the object is identified by performing a matching process on that rectangular area with a sample image of the object, or by performing deep learning processing using the rectangular area as input.

However, the shape (outline) of an object is not necessarily rectangular. When detecting an area where an object is thought to be located as a rectangle, parts of other objects may be reflected in that rectangular area. In addition, the background may also be reflected in that rectangular area. Therefore, when such rectangular areas are used as training data for deep learning, images to be processed, or image matching processing, there is an issue that this can cause a decrease in the accuracy of object identification.

Furthermore, if an object is unknown because it has not been learned, subsequent object identification processing will result in an incorrect recognition. However, performing identification processing on such an object that results in an incorrect recognition is a waste of processing, and it is preferable to eliminate it as early as possible. However, this has not been possible with conventional methods. Furthermore, it is difficult to detect an object as an unlearned object in the first place.

In consideration of the above-mentioned problems, the inventors have invented an information processing system that improves the accuracy of identifying objects captured in image information.

A first invention is an information processing system that identifies an object shown in image information, the information processing system having a first learning processing unit that performs machine learning using first annotation data that associates data indicating the outer shape of the object with attributes to create a first learning model, and a recognition processing unit that identifies object identification information of the object shown in the image information, wherein the recognition processing unit uses the image information and the first learning model to identify the outer shape of the object shown in the image information as an outer shape region and output a reliability for the attribute corresponding to the identified outer shape region, and if the reliability of the attribute corresponding to the identified outer shape region is not below a predetermined threshold , identifies one or more pieces of object identification information for the object shown in the identified outer shape region using a second learning model or image matching process that has been machine learned using image information of the identified outer shape region and second annotation data that associates image data of the object with object identification information, and if the reliability of the attribute corresponding to the identified outer shape region is below a predetermined threshold , the information processing system identifies the object shown in the identified outer shape region as an unknown object.

As in the present invention, by identifying object identification information using a learning model trained by machine learning using the first annotation data, the contour area cut out from the image information can be changed from a conventional rectangular area to an area that follows the contour of the object. This reduces the inclusion of other objects or the background, improving the accuracy of object identification.

In addition, if the attributes corresponding to the outline region of an identified object do not satisfy a predetermined condition, for example, if the reliability is lower than a predetermined threshold, the outline region can be identified as a predetermined object, such as an unknown object (e.g., an object that has not been learned), without performing identification processing on the object identification information of the outline region. This makes it possible to detect unknown objects and also reduces the processing load.

In the above-mentioned invention, the first learning processing unit can be configured as an information processing system that performs machine learning through image segmentation using the first annotation data to create the first learning model.

When performing machine learning, image segmentation methods are preferred.

In the above-mentioned invention, the information processing system can be configured as an information processing system having a second learning processing unit that performs machine learning using the second annotation data to create the second learning model.

Because the outline region is made up of an area that represents the outline of an object, it is preferable to identify object identification information by performing processing such as those of these inventions. Furthermore, if the object identification information of the identified object satisfies a predetermined condition, such as if its reliability is lower than a predetermined threshold, then the outline region is identified as a predetermined object, such as an unknown object, regardless of the identified object identification information. This makes it possible to avoid identifying object identification information with low reliability.

In the above-mentioned invention, the recognition processing unit can be configured as an information processing system that identifies object identification information for the object depicted in the outline region by performing image matching processing between image information of the identified outline region and specimen information of the object stored in the specimen information storage unit.

Image matching processing, as in the present invention, may be used to identify the object identification information of the object captured in the outline region.

A fifth invention is an information processing system for identifying an object shown in image information, comprising: a first learning processing unit that performs machine learning using first annotation data that associates data indicating the outline of the object with attributes to create a first learning model; the information processing system uses the image information and the first learning model to identify the outline of the object shown in the image information as an outline region and output a reliability for the attribute corresponding to the identified outline region; if the reliability of the attribute corresponding to the identified outline region is not below a predetermined threshold , the information processing system identifies object identification information of the object shown in the identified outline region using a second learning model or image matching process that has been machine learned using image information of the identified outline region and second annotation data that associates image data of the object with object identification information; and if the reliability of the attribute corresponding to the identified outline region is below a predetermined threshold , the information processing system identifies the object shown in the identified outline region as an unknown object.

A sixth invention is an information processing system for identifying an object shown in image information, the system including: an image information input reception processing unit that receives input of image information obtained by photographing an object; and an object recognition processing unit that identifies object identification information of the object shown in the image from the received input image information or image information obtained by orthogonalizing the image information, wherein the object recognition processing unit identifies the outer shape of the object using a first learning model created by machine learning using first annotation data that associates data indicating the outer shape of an object with attributes, and the received input image information or image information obtained by orthogonalizing the image information. an information processing system that identifies an object in the specified outer region as an unknown object, outputs a reliability for an attribute corresponding to the specified outer region, and , if the reliability of the attribute corresponding to the specified outer region is not below a predetermined threshold , identifies object identification information of the object appearing in the specified outer region using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information, and, if the reliability of the attribute corresponding to the specified outer region is below a predetermined threshold , identifies the object appearing in the specified outer region as an unknown object.

Even with these inventions configured, the same technical effects as the first invention can be obtained.

The first aspect of the present invention can be realized by loading the program of the present invention into a computer and executing it. In other words, this is an information processing program that causes a computer to function as a first learning processing unit that performs machine learning using first annotation data that associates data indicating the outer shape of an object with attributes to create a first learning model, and a recognition processing unit that identifies object identification information of an object shown in image information, wherein the recognition processing unit uses the image information and the first learning model to identify the outer shape of the object shown in the image information as an outer shape region, outputs a reliability for the attribute corresponding to the identified outer shape region, and , if the reliability of the attribute corresponding to the identified outer shape region is not below a predetermined threshold , identifies one or more object identification information of the object shown in the identified outer shape region using a second learning model or image matching process that has been machine learned using the image information of the identified outer shape region and second annotation data that associates image data of the object with object identification information, and, if the reliability of the attribute corresponding to the identified outer shape region is below a predetermined threshold , identifies the object shown in the identified outer shape region as an unknown object.

The fifth invention can be realized by loading the program of the present invention into a computer and executing it. That is, the information processing program causes a computer to function as a first learning processing unit that performs machine learning using first annotation data that associates data indicating the outline of an object with attributes to create a first learning model, the information processing program causing a computer to identify the outline of an object shown in the image information as an outline region using image information and the first learning model, outputting a reliability for the attribute corresponding to the identified outline region, and, if the reliability of the attribute corresponding to the identified outline region is not equal to or less than a predetermined threshold, identifying object identification information of the object shown in the identified outline region using a second learning model or image matching process that has been machine-learned using image information of the identified outline region and second annotation data that associates image data of the object with object identification information, and, if the reliability of the attribute corresponding to the identified outline region is equal to or less than a predetermined threshold, identifying the object shown in the identified outline region as an unknown object.

A sixth aspect of the present invention can be realized by loading the program of the present invention into a computer and executing it. That is, the information processing program causes a computer to function as an image information input reception processing unit that receives input of image information of an object, and an object recognition processing unit that identifies object identification information of the object in the image from the received input image information or image information obtained by orthogonalizing the image information, wherein the object recognition processing unit identifies the outline of the object in the image as an outline region using a first learning model created by machine learning using first annotation data that associates data indicating the outline of the object with attributes, and the received input image information or image information obtained by orthogonalizing the image information. and outputs a reliability for the attribute corresponding to the specified outer region; if the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold , identifies object identification information of the object appearing in the specified outer region using a second learning model or image matching process that is machine learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information ; and if the reliability of the attribute corresponding to the specified outer region is equal to or less than a predetermined threshold , identifies the object appearing in the specified outer region as an unknown object.

By using the information processing system of the present invention, it is possible to improve the accuracy of identifying objects captured in image information.

1 is a block diagram schematically illustrating an example of a configuration of an information processing system according to the present invention. FIG. 2 is a block diagram schematically illustrating an example of a configuration of an object recognition processing unit in the information processing system of the present invention. FIG. 2 is a block diagram schematically illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. 10 is a flowchart showing an example of a learning process in the information processing system of the present invention. 10 is a flowchart showing an example of a processing process for recognition processing in the information processing system of the present invention. FIG. 2 is a diagram schematically illustrating an example of first annotation data. FIG. 10 is a diagram schematically illustrating another example of the first annotation data. FIG. 10 is a diagram schematically illustrating an example of second annotation data. FIG. 10 is a diagram schematically illustrating another example of the second annotation data. FIG. 10 is a diagram showing an example of captured image information. FIG. 10 is a diagram showing another example of captured image information. 11 is a diagram showing an example of image information obtained by normalizing the captured image information of FIG. 10. FIG. 12 is a diagram showing an example of image information obtained by normalizing the captured image information of FIG. 11. FIG. FIG. 10 is a block diagram illustrating an example of a configuration of an information processing system according to a second embodiment. 10 is a diagram showing a schematic diagram of a state in which input specifying a shelf area is received for normal-orientation image information obtained by normalizing image information obtained by photographing a display shelf on which objects are displayed. FIG. 10 is a diagram showing a schematic diagram of a state in which input specifying a shelf area is received for normal-orientation image information obtained by normalizing image information captured of a display shelf on which objects are hung and displayed. FIG. FIG. 10 is a diagram showing an example of a case where an outer shape area is identified from image information of a shelf area. 10 is a flowchart illustrating an example of a processing process of a recognition process in an information processing system according to a second embodiment. FIG. 10 is a diagram showing an example of captured image information. FIG. 20 is a diagram showing an example of normalized image information obtained by performing normalization processing on the photographed image information of FIG. 19 . FIG. 10 is a block diagram illustrating an example of the configuration of an information processing system according to a third embodiment. FIG. 4 is a diagram illustrating an example of specimen information stored in a specimen information storage unit. FIG. 10 is a block diagram illustrating an example of an object recognition processing unit according to a fourth embodiment.

An example of the processing functions of the information processing system 1 of the present invention is shown in block diagrams in Figures 1 and 2. The information processing system 1 uses a management terminal 2 and an image information input terminal 3. Figure 1 is a block diagram showing the overall functions of the information processing system 1, and Figure 2 is a block diagram showing the functions of the object recognition processing unit 213, which will be described later.

The management terminal 2 is a computer used by an organization such as a company that operates the information processing system 1. The image information input terminal 3 is a terminal that inputs image information containing objects to be identified, such as image information captured of a store display shelf displaying the products to be identified.

The management terminal 2 and image information input terminal 3 in the information processing system 1 are realized using a computer. Figure 3 shows a schematic diagram of an example of the hardware configuration of a computer. The computer has a computing device 70 such as a CPU that executes the arithmetic processing of a program, a storage device 71 such as RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or mouse that can input information, and a communication device 74 that sends and receives the processing results of the computing device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

If the computer is equipped with a touch panel display, the display device 72 and input device 73 may be integrated. Touch panel displays are often used in portable communication devices such as tablet computers and smartphones, but are not limited to these.

A touch panel display is a device that combines the functions of a display device 72 and an input device 73, in that input can be made directly on the display using a specified input device (such as a touch panel pen) or a finger.

In addition to the devices described above, the image information input terminal 3 may also be equipped with a camera or other imaging device. Portable communication terminals such as mobile phones, smartphones, and tablet computers can also be used as the image information input terminal 3.

The functions of the various means of the present invention are only logically distinct, and may physically or practically be in the same area. The order of processing performed by the various means of the present invention may be changed as appropriate. Part of the processing may also be omitted. For example, the normalization processing described below may be omitted. In this case, processing can be performed on image information that has not been normalized.

The information processing system 1 has a learning processing unit 20, a recognition processing unit 21, and an object information storage unit 22. The learning processing unit 20 has a first learning processing unit 201 and a second learning processing unit 202.

The first learning processing unit 201 uses the first annotation data to perform a machine learning learning process, preferably a learning process using an image segmentation method, on image information containing the object to be identified, such as image information captured of a display shelf displaying the product to be identified. This learning process is a learning process in machine learning, and performs a learning process using image segmentation to create a learning model using, for example, deep learning.

The first annotation data is data that associates the contour of an object that may be the target of identification as its outer shape, data in which the inside of the closed area formed by that contour is masked, with tags (labels) that classify the attributes of the object's contour. Note that the outer shape may be the object's contour itself, or a shape that includes the object's contour and follows the object's contour, or a shape that has some width from the object's contour, such as a shape that indicates the object's contour. In other words, the outer shape of an object's contour is not limited to a rectangular area, and may also be a non-rectangular area that reflects the object's contour.

For example, if the object is a product displayed on a display shelf, the first annotation data is data in which the outline of the object that may be displayed on the display shelf is used as the outer shape, and the inside of the closed area formed by the outline is masked, and a tag (label) that classifies the attributes of the object's outline is associated with the data.

There does not have to be one piece of first annotation data per object; there can be multiple pieces of first annotation data per object. In other words, data obtained by masking the interior of an object's outline from multiple directions can be associated with attributes, and each piece of first annotation data for the object can be used. Attributes refer to the type of object. Object types include animal types (e.g., elephant, tiger, lion), bird types (e.g., chicken, parakeet, peacock), insect types (e.g., rhinoceros beetle, stag beetle, butterfly), fish types (e.g., tuna, mackerel, sardine), plant types (e.g., cherry blossom, plum, lily, rose, chrysanthemum, tulip), and mobile object (e.g., vehicle, aircraft) types (e.g., automobile, bicycle, motorcycle, airplane, helicopter, drone, UAV (Unmanned Aircraft Vehicle), ship, boat), as well as various other types, such as the classification of the object's container and the object's object identification information (e.g., JAN code). Container classifications may be by type, such as cans, bottles, boxes, or pouches, or may be further subdivided according to use, such as detergent containers. In other words, attributes only need to indicate how the closed area defined by the contour of the object is classified. Examples of the first annotation data are shown in Figures 6 and 7. Figures 6 and 7 show the case of displayed products as objects, but as mentioned above, this is not a limitation. Object identification information is not limited to JAN codes, and may be any information that can uniquely identify an object.

Figure 6 shows the first annotation data obtained by associating the outline of a can and the masked area inside the closed area formed by the outline with the attribute "can," while Figure 7 shows the first annotation data obtained by associating the outline of a shampoo refill and the masked area inside the closed area formed by the outline with the attribute "pouch container." As an attribute in the first annotation data, object identification information such as a JAN code may be used instead of the container classification if the object can be identified from the object's outline itself.

By performing a learning process for machine learning using the first annotation data in the first learning processing unit 201, a learning model (first learning model) for identifying the contour area of an object is created from image information of an object that may be the target of identification. For example, a learning model (first learning model) for identifying the contour area of an object is created from image information of an image of a display shelf. Note that the first learning model may be a learning model that can identify the attributes of an object in addition to the contour area of the object.

The second learning processing unit 202 uses the second annotation data to perform machine learning learning processing, thereby creating a learning model (second learning model) for identifying object identification information of an object in a specified image information, preferably image information of an outline region described below, from that region. The learning processing in this case is preferably performed using an image classification method, but methods such as object detection, image classification/object localization, etc. may also be used. The second learning model may be a learning model that allows attributes corresponding to the outline region, in addition to image information of the outline region, to be input as input values for the second learning model.

The second annotation data is data that associates image information of an object that may be identified with the object identification information of that object as a tag (label). For example, if the object is a displayed product, the data associates image information of the product that may be displayed on a shelf with the object identification information of that object as a tag (label). If the object is an animal, the data associates image information of the animal that may be identified with the animal's name or other identification information (scientific name, common name, etc.) as a tag (label) of the object identification information. Examples of second annotation data are shown in Figures 8 and 9. While Figures 8 and 9 show the case of a displayed product as the object, as mentioned above, this is not a limitation. Like the first annotation data, the second annotation data does not have to be one per object; multiple pieces of data may be present. In other words, the object may be photographed from multiple directions, and the image information of the object from each direction may be associated with the object identification information to create the second annotation data. Furthermore, if attributes corresponding to the outer shape region can also be input as input values in the second learning model, the second annotation data may be data that associates image information of an object that may be the subject of identification with the attributes of that object and the object identification information of that object as tags (labels).

Figure 8 shows the case where image information of a can is associated with object identification information to form second annotation data, and Figure 9 shows the case where image information of a shampoo refill is associated with object identification information to form second annotation data. The second annotation data in Figure 8 corresponds to the first annotation data in Figure 6, and the second annotation data in Figure 8 corresponds to the second annotation data in Figure 7.

The recognition processing unit 21 has an image information input reception processing unit 210, an image information storage unit 211, an image information alignment processing unit 212, and an object recognition processing unit 213.

The image information input reception processing unit 210 receives input of image information (captured image information) including the object to be identified, captured by the image information input terminal 3, and stores it in the image information storage unit 211 (described below). It is preferable to receive input from the image information input terminal 3, including the captured image information, the date and time of capture, and image information identification information that identifies the image information. An example of captured image information when the object is a product displayed on a display shelf is shown in Figures 10 and 11. Figures 10 and 11 show captured image information of a display shelf with three shelves on which the object is displayed. While this invention does not specifically specify the processing, display shelves and shelves are often long horizontally. Therefore, in this processing, the image may be divided into sections at a certain width and used as the processing target for each process.

The image information storage unit 211 stores captured image information, shooting date and time, image information identification information, etc. received from the image information input terminal 3, in association with each other. Captured image information may be any image information that is the subject of processing in the present invention. Generally, when simply capturing an image, it is difficult to capture the subject facing forward, so it is advisable to perform a correction process to correct the image to a forward facing state, such as keystone correction. This also includes image information in which multiple images of a single subject are taken and then combined into a single image. Image information after distortion correction has been performed is also included in captured image information.

The image information orientation processor 212 performs a process (orthogonalization process) on the captured image information stored in the image information storage unit 211 to correct the captured object so that it is facing upright, for example, by performing keystone correction, to generate orthogonalized image information. Keystone correction is a correction process performed to orient the object captured in the captured image information so that it is facing upright. For example, if the object is a displayed commodity, this is a correction process performed so that the shelves on which the displayed commodity is displayed are horizontal. Orthogonalization is the process of transforming the image information so that the optical axis of the lens of the image capture device is aligned perpendicular to the plane of the object being captured, so that it appears the same as if it were captured from a sufficiently distant location; for example, this can be done by keystone correction, but is not limited to this.

The keystone correction process performed by the image information alignment processor 212 accepts input specifying four vertices in the captured image information, and performs keystone correction using each of those vertices. If the object is a commodity displayed on a display shelf, the four vertices that are accepted for designation may be the four vertices of the shelf level on the display shelf, or the four vertices of the shelf position on the display shelf. They may also be the four vertices of a group of two or three shelf levels. Any four points can be specified as the four vertices. Figure 12 shows an example of captured image information (aligned image information) obtained by orthogonally aligning the captured image information of Figure 10, and Figure 13 shows an example of captured image information (aligned image information) obtained by orthogonally aligning the captured image information of Figure 11.

The object recognition processing unit 213 performs processing to recognize objects captured in image information, preferably captured image information or upright image information.

The object recognition processing unit 213 has an outer shape specification processing unit 2131 and an object identification processing unit 2132.

The contour identification processing unit 2131 identifies the contour area (contour area) of an object captured in the upright image information.

The contour identification processing unit 2131 inputs the captured image information and upright image information as input values to the learning model (first learning model) trained by the first learning processing unit 201, and identifies the contour region from the input image information. That is, the contour identification processing unit 2131 inputs the region to be processed, such as upright image information, to the learning model trained by the first learning processing unit 201 (a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized), and identifies the contour region based on the output value. The identified contour region is assigned contour identification information that identifies the contour region, and stored in the object identification information storage unit 22 together with position information in the captured image information and upright image information (for example, coordinates in the image information).

The output value output by the contour identification processing unit 2131 should also include the attribute tag corresponding to the contour region and its reliability (probability). In this case, the attribute tag with the highest reliability is identified as the tag for the contour region, and if the reliability is below a certain threshold, the identified contour region is identified as a specified object, such as an unknown object (an object that cannot be identified for reasons such as not being learned).

The object identification processing unit 2132 inputs the image information of the outline region as an input value to a learning model (second learning model) trained by the second learning processing unit 202, and identifies the identification information of the object in that region from the input image information. That is, the object identification processing unit 2132 inputs the image information of the outline region to be processed into the learning model trained by the second learning processing unit 202 (a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized), and identifies the object identification information of the object in the outline region based on the output value. At this time, in addition to the image information of the outline region, tag information of the attribute corresponding to the outline region identified by the outline identification processing unit 2131 may also be input as an input value. Note that at this time, one object identification information or multiple object identification information may be identified for the outline region. If multiple object identification information is identified, candidates for the object identification information to be identified are output.

The output value output by the object identification processing unit 2132 also includes the reliability (probability) of the object identification information identified by that outline region. At this time, the object identification information with the highest reliability is identified as the object identification information for that outline region, but if the highest reliability is below a certain threshold, the input outline region is identified as an unknown object.

The object recognition processing unit 213 may perform the processing of the contour specification processing unit 2131 and the object identification processing unit 2132 together using deep learning or the like.

The object identification information storage unit 22 stores information indicating the object identification information of objects shown in the photographed image information and upright image information. For example, the object identification information is associated with the photographing date and time information, image information identification information of the photographed image information, image identification information of the upright image information, and external shape identification information for identifying the external shape, and these are stored in the object identification information storage unit 22.

Next, an example of the processing process of the information processing system 1 of the present invention will be explained using the flowcharts in Figures 4 and 5.

First, the learning process for learning the learning model used in the recognition processing unit 21 of the information processing system 1 of the present invention will be explained using the flowchart in Figure 4.

First annotation data is created as training data for the learning model in the first learning processing unit 201 (S100). The first annotation data is image data in which the outline of an object that may be the target of identification is formed, and the closed area inside that outline is masked. Attributes are associated with this image data as tags.

Similarly, second annotation data is created as training data for the learning model in the second learning processing unit 202 (S110). The second annotation data is created by associating an image of an object that may be the target of identification with the object identification information of that object as a tag. The image information of the object in this case preferably corresponds to the first annotation data, and is preferably image information that shows the outline of the object.

Then, the created first annotation data is input as training data, and a learning process for machine learning is performed in the first learning processing unit 201, creating a learning model (first learning model) for identifying the contour area of the object (S120).

The created second annotation data is input as training data, and a learning process for machine learning is performed in the second learning processing unit 202 to create a learning model (second learning model) for identifying object identification information of an object in the image information, preferably the outline area, from that area (S130).

By performing the above process, each learning model can be created.

Next, the recognition process for identifying object identification information of an object captured in image information of the object to be identified will be explained using the flowchart in Figure 5.

Photographed image information of an object to be identified is input from the image information input terminal 3, and this input is accepted by the image information input acceptance processing unit 210 of the management terminal 2 (S200). It also accepts input of the shooting date and time and image information identification information of the photographed image information. The image information input acceptance processing unit 210 then associates the accepted input photographed image information, shooting date and time, and image information identification information of the photographed image information and stores them in the image information storage unit 211.

When a specified operation input is received on the management terminal 2, the image information alignment processing unit 212 extracts the captured image information stored in the image information storage unit 211, receives input of four points for performing alignment processing such as keystone correction processing, and executes the alignment processing (S210).

Then, the management terminal 2 receives a predetermined operational input for the upright image information, and executes a process to identify the outer shape (S220). That is, the outer shape identification processing unit 2131 inputs some or all of the area in the captured image information or upright image information as input values into the learning model (first learning model) trained by the first learning processing unit 201, and identifies the outer shape area from the input image information.

As described above, the contour regions of each object captured in the upright image information are identified, and their attributes (tags) are identified. If the reliability of the identified tag is below a certain threshold (S230), the object in that contour region is identified as an unknown object (S240).

On the other hand, if the reliability of the identified tag is greater than a certain threshold (S230), the object identification processing unit 2132 inputs the image information of the outline region as an input value into the learning model (second learning model) trained in the second learning processing unit 202, and identifies the object identification information of the object appearing in the outline region (S250). If the reliability of the identified object identification information is equal to or less than a certain threshold (S260), the object in the outline region is identified as an unknown object (S240).

On the other hand, if the reliability of the identified object identification information is greater than a certain threshold (S260), the identified object identification information is stored in the object identification information storage unit 22 in association with the shooting date and time, image information identification information of the captured image information, image information identification information of the upright image information, and external shape identification information.

In addition, input regarding the process of correcting the identified object identification information may also be accepted.

By performing the above processing, it is possible to identify the object identification information of objects captured in captured image information. Furthermore, unlike conventional systems, the outer region is not a rectangular region, but rather the object is identified along the outline of the object's contour. This eliminates unnecessary information contained in the outer region, such as noise from other objects, thereby improving recognition accuracy.

Note that when object identification information is used as an attribute in the first annotation data (when an object can be identified from its external shape), the object identification information of the object can be identified by inputting the image information into the first learning model and identifying the external shape region. In this case, it is not necessary to perform processing by the second learning processing unit 202 and the object identification processing unit 2132; once the external shape region is identified by the external shape identification processing unit 2131, the object identification information of the object depicted in that external shape region can be identified using the object identification information of the attribute as an output result from the first learning model.

In the information processing system 1 of Example 1 described above, in the case of displayed products displayed on a display shelf as objects, the processing may be further performed for each shelf area (shelf area). An example of the processing functions of the information processing system 1 in this case is shown in a block diagram in Figure 14.

In the information processing system 1 of this embodiment 2, the recognition processing unit 21 further includes a shelf level identification processing unit 214.

The shelf level identification processing unit 214 identifies the area of a shelf on a display shelf (shelf level area) where an object may be placed, from the upright image information obtained by performing keystone correction processing on the captured image information in the image information alignment processing unit 212. The captured image information and upright image information show a display shelf, but the display shelf has a shelf level area where objects are displayed. Therefore, the shelf level area is identified from the upright image information. To identify the shelf level area, the operator of the management terminal 2 may manually specify the shelf level area, which the shelf level identification processing unit 214 may accept, or the shelf level area may be identified automatically from the second time onwards based on the shelf level area information manually input the first time.

Figure 15 shows a schematic diagram of a state in which input for specifying a shelf area has been received for upright image information obtained by orthogonally ...

When identifying shelf level areas, the shelf level identification processing unit 214 may use deep learning to identify shelf level areas. In this case, the above-mentioned upright image information may be input into a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the shelf level area may be identified based on the output values. Furthermore, the learning model may be one in which shelf level areas are assigned as correct answer data to various upright image information.

The shelf level area identified by the shelf level identification processing unit 214 identifies its image information as shelf level area image information. The shelf level identification processing unit 214 may actually cut out the image information, or it may virtually cut out the image information of the area using coordinates or the like without actually cutting out the image information. If the display shelf has multiple shelves, each is cut out as shelf level area image information. The coordinates indicating the shelf level area are the coordinates of the vertices used to identify the area, and may be, for example, four points in the upright image information, such as the coordinates of the upper right and lower left, or two points in the upper left and lower right. They are also relative coordinates based on a predetermined location in the image information (for example, the upper left vertex of the display shelf), such as the display shelf in the upright image information. In this specification, cutting out image information may mean actually cutting out the image information, as in the case of cutting out by the shelf level identification processing unit 214, or it may mean virtually cutting out the image information of the area using coordinates or the like without actually cutting out the image information.

The contour specification processing unit 2131 and object identification processing unit 2132 in the object recognition processing unit 213 in the information processing system 1 of this embodiment may perform the following processing in addition to the processing of Example 1.

The outer shape identification processing unit 2131 identifies the outer shape area (outer shape area) for each shelf in the shelf area in the upright image information. The outer shape is the area where an object is placed, regardless of whether the object is placed there or not. The size of the outer shape area is the same as or approximately the same as the size of the object to be placed there.

The contour identification processing unit 2131 inputs the captured image information, upright image information, or image information of the shelf area as input values to the learning model (first learning model) trained by the first learning processing unit 201, and identifies the contour area from the input image information. That is, the contour identification processing unit 2131 inputs image information of the area to be processed, for example, the shelf area, to the learning model trained by the first learning processing unit 201 (a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized), and identifies the contour area based on the output value. The identified contour area is assigned contour identification information that identifies the contour area, and stored in the object identification information storage unit 22 together with position information (for example, coordinates in the image information) in the captured image information, upright image information, or image information of the shelf area.

The output value output by the contour identification processing unit 2131 includes the attribute tag corresponding to the contour region, for example, can, bottle, pouch container, etc., along with its reliability (probability). At this time, the attribute tag with the highest reliability is identified as the tag for the contour region, and if that reliability is below a certain threshold, the identified contour region is identified as an unknown object.

Figure 17 shows an example of an outline region identified from image information of a shelf level region. Figure 17(a) is an example of image information of a shelf level region input to a learning model trained by the first learning processing unit 201, and Figure 17(b) is a diagram showing an example of an outline region identified using the learning model in the image information of the shelf level region used as an input value in Figure 17(a). Figure 17(b) shows an example of an outline region identified in a shelf level region superimposed, but the image information of the identified outline region may also be cut out and output as is.

The object identification processing unit 2132 inputs the image information of the outline region as an input value to a learning model (second learning model) trained by the second learning processing unit 202, and identifies the identification information of the object in that region from the input image information. That is, the object identification processing unit 2132 inputs the image information of the outline region to be processed into the learning model trained by the second learning processing unit 202 (a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized), and identifies the object identification information of the object in the outline region based on the output value. At this time, in addition to the image information of the outline region, attribute tags corresponding to the outline region identified by the outline identification processing unit 2131, such as information on cans, bottles, and pouch containers, may be input as input values. Note that one or more object identification information may be identified for the outline region. If multiple object identification information is identified, candidates for object identification information to be identified are output.

The output value output by the object identification processing unit 2132 also includes the reliability (probability) of the object identification information identified by that outline region. At this time, the object identification information with the highest reliability is identified as the object identification information for that outline region, but if the highest reliability is below a certain threshold, the input outline region is identified as an unknown object.

Similar to Example 1, the object recognition processing unit 213 may perform the processing of the outer shape specification processing unit 2131 and the object identification processing unit 2132 together using deep learning or the like.

The object identification information storage unit 22 stores information indicating the object identification information of the objects displayed on the outer shape of each shelf of the display shelf. For example, the object identification information is associated with the photographing date and time information, store information, image information identification information of the photographed image information, image identification information of the upright image information, and outer shape identification information for identifying the outer shape, and these are stored in the object identification information storage unit 22.

Next, an example of the processing process of the information processing system 1 in this embodiment will be described using the flowcharts in Figures 4 and 18. Note that the following description of this embodiment will focus on the case where object identification information for displayed objects is identified from captured image information.

Note that the learning process (Figure 4) for learning the learning model used by the recognition processing unit 21 of the information processing system 1 in this embodiment is the same as in Example 1, so a description thereof will be omitted.

The recognition process for identifying object identification information of objects displayed on a display shelf from image information captured of the display shelf is explained using the flowchart in Figure 18.

Photographed image information of store display shelves is input from the image information input terminal 3, and this input is accepted by the image information input acceptance processing unit 210 of the management terminal 2 (S300). Figure 19 shows an example of photographed image information. It also accepts input of the date and time of photography, store identification information, and image information identification information of the photographed image information. The image information input acceptance processing unit 210 then associates the accepted input photographed image information, date and time of photography, store identification information, and image information identification information of the photographed image information, and stores them in the image information storage unit 211.

When a specified operation input is received at the management terminal 2, the orthogonal image information orthogonalization processing unit 212 extracts the captured image information stored in the image information storage unit 211, receives input of four shelf positions (display shelf positions) that are the vertices for performing orthogonalization processing such as keystone correction, and executes the orthogonalization processing (S310). Figure 20 shows an example of captured image information (orthogonal image information) that has been orthogonalized in this way.

Then, by receiving a specified operation input on the management terminal 2 in response to the normal image information, the shelf level identification processing unit 214 identifies the shelf level position area (S320). In other words, it accepts input of the shelf level area in the normal image information. Figures 15 and 16 show the state in which the shelf level area has been identified from the normal image information.

Once the shelf level area is identified in this manner, image information of the shelf level area is extracted from the upright image information. Then, a process is performed to identify the outline for each shelf level in the shelf level area image information (S330). That is, the outline identification processing unit 2131 inputs the image information of the shelf level area as an input value into the learning model (first learning model) trained by the first learning processing unit 201, and identifies the outline area from the input image information.

As described above, the contour regions of each object captured in the upright image information are identified, and their attributes (tags) are identified. If the reliability of the identified tag is below a certain threshold (S340), the object in that contour region is identified as an unknown object (S350).

On the other hand, if the reliability of the identified tag is greater than a certain threshold (S340), the object identification processing unit 2132 inputs the image information of the outline region as an input value into the learning model (second learning model) trained in the second learning processing unit 202, and identifies the object identification information of the object appearing in the outline region (S360). If the reliability of the identified object identification information is equal to or less than a certain threshold (S370), the object in the outline region is identified as an unknown object (S350).

On the other hand, if the reliability of the identified object identification information is greater than a certain threshold (S370), the identified object identification information is stored in the object identification information storage unit 22 in association with the photographing date and time, store identification information, image information identification information of the photographed image information, image information identification information of the upright image information, and external shape identification information.

Note that it is not always possible to identify object identification information for all outer shape regions. Therefore, for outer shape regions that cannot be identified, input of object identification information is accepted, and the accepted input object identification information is stored in the object identification information storage unit 22 in association with the shooting date and time, store identification information, image information identification information for the captured image information, image information identification information for the upright image information, and outer shape identification information. Input may also be accepted similarly for the process of correcting identified object identification information.

By performing the above processing, it is possible to identify the object identification information of the objects displayed on the shelves of the display shelf captured in the captured image information. Furthermore, unlike conventional systems, the outer shape region is not a rectangular region, but rather the object is identified along the outline of the object's contour. This eliminates unnecessary information contained in the outer shape region, such as noise from other objects, thereby improving recognition accuracy.

Note that when object identification information is used as an attribute in the first annotation data (when an object can be identified from its external shape), the object identification information of the object can be identified by inputting the image information into the first learning model and identifying the external shape region. In this case, it is not necessary to perform processing by the second learning processing unit 202 and the object identification processing unit 2132; once the external shape region is identified by the external shape identification processing unit 2131, the object identification information of the object depicted in that external shape region can be identified using the object identification information of the attribute as an output result from the first learning model.

In Examples 1 and 2, configurations were described in which machine learning was used in two processes: specifying the outline region and identifying object identification information from the outline region. However, image matching processing may be used for the process of identifying object identification information from the outline region. An example of the configuration of information processing system 1 in this case is shown in Figure 21. Note that Figure 21 shows a case based on the configuration of information processing system 1 in Example 2, but it goes without saying that it may also be based on Example 1.

In the information processing system 1 of this embodiment, the learning processing unit 20 does not need to have a second learning processing unit 202. In addition, the recognition processing unit 21 is equipped with a specimen information storage unit 215 that stores specimen information used in the image matching process.

The specimen information storage unit 215 stores specimen information for identifying the object displayed on the shelf of the display shelf shown in the image information. The specimen information is image information of objects that may be displayed on the display shelf, photographed from multiple angles, such as from above, below, left, right, or diagonally. Figure 22 shows an example of specimen information stored in the specimen information storage unit 215. Figure 22 shows specimen information of canned beer photographed from various angles, but is not limited to canned beer. The specimen information storage unit 215 stores specimen information in association with object identification information.

The specimen information storage unit 215 may store, together with or instead of the specimen information, information extracted from the specimen information that is required for calculating similarity, such as information on pairs of image features and their positions. The specimen information also includes information required for calculating similarity. In this case, when performing matching processing between image information of the outline region (described below) and the specimen information, the object recognition processing unit 213 does not need to calculate image features for the specimen information every time, thereby reducing calculation time.

The specimen information stored in the specimen information storage unit may also be image information of an object in which the outer shape of the object's contour in the first annotation data used during the learning process by the first learning processing unit 201 has been masked. That is, when creating the first annotation data, image information of the object photographed from one or more directions or its image features is used as specimen information. Then, from the image information of the photographed object, the contour is used as the outer shape, and the inside of the closed area is masked, and attributes are tagged to create the first annotation data. Through this process, specimen information and first annotation data can be created together.

In this embodiment, the object identification processing unit 2132 performs a matching process between the image information of the contour region identified by the contour identification processing unit 2131 and the specimen information stored in the specimen information storage unit 215, and identifies the object identification information of the object displayed in that contour region. That is, the image information of the contour region of a certain shelf (the contour identification information of this contour region is designated X) is used to calculate image features from each of the specimen information stored in the specimen information storage unit, and similarity is determined by finding pairs of feature points. The most similar specimen information is then identified, and if the similarity at that time is equal to or greater than a predetermined threshold, the object identification information corresponding to that specimen information is identified based on the specimen information storage unit 215. The identified object identification information is then used as the object identification information of the object displayed in the contour of the contour identification information X. For contours determined to be unsimilar to any specimen information, information indicating that the contour identification information is "empty" (information indicating that no object is present) is added to the contour identification information. The object identification processing unit 2132 stores the identified object identification information or information indicating that the object is "empty" in the object identification information storage unit 22 in association with the image capture date and time, store identification information, image information identification information of the captured image information, image information identification information of the upright image information, and external shape identification information.

As an example, the object identification processing unit 2132 specifically executes the following process. First, it determines the similarity between the image information composed of the coordinates of the outline region to be processed and the specimen information stored in the specimen information storage unit 215, identifies the object identification information corresponding to the specimen information with the highest similarity, and if the identified similarity is equal to or greater than a predetermined threshold, identifies it as the object identification information of the object displayed in the outline region composed of the coordinates.

To determine the similarity between the image information of the outline and the specimen information, the following processing is performed. First, in the processing prior to the identification processing of the object identification information in the object identification processing unit 2132, the image information of the outline region on the shelf in the upright image information and the specimen information are oriented in the same direction (not turned sideways or upside down), and the sizes of the respective image information are roughly the same (if the sizes of the image information differ by more than a predetermined range, the sizes of the respective image information are adjusted so that they are within the predetermined range before determining the similarity).

To determine the similarity between the image information of the outline region and the specimen information, the object identification processing unit 2132 extracts feature points based on the image features (e.g., local features) of the image information of the outline and feature points based on the image features (e.g., local features) of the specimen information. It then detects the most similar pair of feature points in the image information of the outline and feature points in the specimen information, and calculates the difference in coordinates of corresponding points between them. It then calculates the average difference. The average difference indicates the overall average amount of movement between the image information of the outline region and the specimen information. It then compares the coordinate differences of all feature point pairs with the average coordinate difference, and excludes pairs with a high degree of discrepancy. It then ranks the similarity based on the number of remaining corresponding points.

The above method can be used to calculate the similarity between the image information of the outline region and the specimen information. Furthermore, to improve accuracy, the EMD (Earth Movement Distance) between the color histograms can be calculated and used as a measure of similarity. This allows for a comparison of similarities that are relatively resistant to environmental changes, such as brightness information in the captured image information, allowing for highly accurate identification.

Another way to determine similarity is to calculate the EMD between the signatures (a collection of image features and weights) of the image information of each contour region and use this as a measure of similarity. For example, the image feature of the signature can be determined by calculating the frequency distribution in the HSV color space of the image information of the contour region, grouping it based on hue and saturation, and using this as the image feature based on the number of features and the area in the HSV color space. Grouping based on hue and saturation is done to reduce dependence on lightness so that it is not significantly affected by shooting conditions.

In addition, to speed up processing, similarities such as L2 distance between image features such as color correlograms and color histograms of image information within an appropriate color space can be used instead of signatures and EMD.

The similarity determination is not limited to the above. The identified object identification information is stored in the object identification information storage unit 22 in association with the photographing date and time information, store information, image information identification information of the photographed image information, image identification information of the upright image information, and external shape identification information.

For contours for which object identification information could not be identified, information indicating that the contour area is "empty" (information indicating that the object is not missing, etc.) is stored in the object identification information storage unit 22.

As described above, even when using image matching processing to identify an object from image information of its outline region, the image matching processing can be performed with high accuracy because the outline region is not a rectangular region.

As a modification of the above-described second or third embodiment, a shelf level comparison processing unit 2143 is provided to detect changes in shelf levels, and if there are no changes in shelf levels, the previous recognition results can be used as is. An example of the object recognition processing unit 213 in this case is shown in Figure 23.

The shelf level comparison processing unit 2133 determines that the object identification information for each external shape of a shelf level in the previous (N-1) upright image information or the "empty" symbol is the same if there is a high degree of similarity between the image information for the shelf level area in the previous (N-1) upright image information and the image information for the shelf level area in the current (N) upright image information. As described above, this similarity determination process may determine similarity based on the image feature values of the image information for the shelf level area in the previous (N-1) upright image information and the image information for the shelf level area in the current (N) upright image information, or may use the EMD between color histograms. This method is not limited to these. Then, rather than performing a process for identifying each external shape unit in the object identification processing unit 2132, the object identification processing unit 2132 stores the object identification information for each external shape of a shelf level in the Nth upright image information in the object identification information storage unit 22 as the same as the object identification information for each external shape of the same shelf level in the N-1th upright image information. This allows us to omit processing for shelves that rarely change, such as shelves with little object movement or shelves that are managed in extremely short cycles.

The processes of Examples 1 to 4 described above can also be combined as appropriate. Furthermore, the order of each process is not limited to the order described in the present specification, and can be changed as appropriate to the extent that the objective is achieved. Furthermore, although the processing in the object recognition processing unit 213 is performed on upright image information obtained by performing a normalization process on the photographed image information, it may also be performed on the photographed image information. In this case, upright image information should be read as photographed image information.

In addition, in Examples 2 and 4, instead of identifying the shelf level area in the recognition processing unit 21 and then identifying the outline area (described below) from that, it is also possible to configure the recognition processing unit 21 to identify the outline area (described below) from the entire captured image information, upright image information, or image information of the shelf level area without identifying the shelf level area. In this case, the shelf level identification processing unit 214 does not need to be provided, and the processing may be configured not to be performed.

In the above-described embodiments, examples have been given of display shelves in convenience stores, supermarkets, etc., but the present invention is not limited to these. For example, the present invention can be applied to medicines (objects) displayed on display shelves (medicine shelves) in pharmacies. Similarly, the present invention can be applied to objects displayed on display shelves in warehouses.

The object may also be something other than merchandise displayed on a display shelf. For example, the object may be any type of object that can be identified, such as an animal, bird, insect, fish, plant, or moving object. In this way, image information captured of various objects may be applied to image information other than merchandise displayed on a display shelf.

By using the information processing system 1 of the present invention, it is possible to improve the accuracy of identifying objects when identifying displayed objects from image information.

DESCRIPTION OF SYMBOLS 1: Information processing system 2: Management terminal 3: Image information input terminal 20: Learning processing unit 21: Recognition processing unit 22: Object identification information storage unit 70: Arithmetic unit 71: Storage device 72: Display device 73: Input device 74: Communication device 201: First learning processing unit 202: Second learning processing unit 210: Image information input reception processing unit 211: Image information storage unit 212: Image information alignment processing unit 213: Object recognition processing unit 214: Shelf level identification processing unit 2131: Outline identification processing unit 2132: Object identification processing unit 2133: Shelf level comparison processing unit

Claims (9)

An information processing system for identifying an object appearing in image information,
a first learning processing unit that performs machine learning using first annotation data in which data indicating the external shape of an object is associated with attributes, to create a first learning model;
a recognition processing unit that identifies object identification information of an object shown in the image information;
It has
The recognition processing unit
Using the image information and the first learning model, an outline of an object shown in the image information is identified as an outline region, and a reliability of an attribute corresponding to the identified outline region is output;
If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, one or more pieces of object identification information of the object appearing in the specified outer region are identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information of the object;
If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object captured in the specified outline region is identified as an unknown object.
An information processing system comprising: The first learning processing unit
performing machine learning by image segmentation using the first annotation data to create the first learning model;
2. The information processing system according to claim 1, wherein: The information processing system includes:
a second learning processing unit that performs machine learning using the second annotation data to create the second learning model;
3. The information processing system according to claim 1, further comprising: The recognition processing unit
performing an image matching process between the image information of the specified outline region and specimen information of the object stored in a specimen information storage unit, thereby identifying object identification information of the object shown in the outline region;
3. The information processing system according to claim 1 or 2. An information processing system for identifying an object appearing in image information,
a first learning processing unit that performs machine learning using first annotation data in which data indicating the external shape of an object is associated with attributes, to create a first learning model;
It has
Using the image information and the first learning model, an outline of an object shown in the image information is identified as an outline region, and a reliability of an attribute corresponding to the identified outline region is output;
If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, the object identification information of the object shown in the specified outer region is identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information;
If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object shown in the specified outline region is identified as an unknown object.
An information processing system comprising: An information processing system for identifying an object appearing in image information,
an image information input reception processing unit that receives input of image information obtained by photographing an object;
an object recognition processing unit that identifies object identification information of an object shown in the image information from the received input image information or image information obtained by orthogonally positioning the image information;
It has
The object recognition processing unit
using a first learning model created by machine learning using first annotation data in which data indicating the outline of an object is associated with an attribute, and the received input image information or image information obtained by orthogonally positioning the image information, to identify the outline of the object as an outline region, and output a reliability for the attribute corresponding to the identified outline region;
If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, the object identification information of the object shown in the specified outer region is identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information;
If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object captured in the specified outline region is identified as an unknown object.
An information processing system comprising: Computer,
a first learning processing unit that performs machine learning using first annotation data in which data indicating the external shape of an object is associated with attributes, to create a first learning model;
a recognition processing unit that identifies object identification information of an object shown in the image information;
An information processing program that functions as
The recognition processing unit
Using the image information and the first learning model, an outline of an object shown in the image information is identified as an outline region, and a reliability of an attribute corresponding to the identified outline region is output;
If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, one or more pieces of object identification information of the object appearing in the specified outer region are identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information of the object;
If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object captured in the specified outline region is identified as an unknown object.
An information processing program characterized by: Computer,
a first learning processing unit that performs machine learning using first annotation data in which data indicating the external shape of an object is associated with attributes, to create a first learning model;
An information processing program that functions as
Using image information and the first learning model, an outline of an object shown in the image information is identified as an outline region, and a reliability for an attribute corresponding to the identified outline region is output;
If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, the object identification information of the object shown in the specified outer region is identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information;
If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object shown in the specified outline region is identified as an unknown object.
An information processing program characterized by: Computer,
an image information input reception processing unit that receives input of image information obtained by photographing an object;
an object recognition processing unit that identifies object identification information of an object shown in the image information from the received input image information or image information obtained by orthogonally positioning the image information;
An information processing program that functions as
The object recognition processing unit
using a first learning model created by machine learning using first annotation data in which data indicating the outline of an object is associated with an attribute, and the received input image information or image information obtained by orthogonally positioning the image information, to identify the outline of the object as an outline region, and output a reliability for the attribute corresponding to the identified outline region;
If the reliability of the attribute corresponding to the specified outer region is not equal to or less than a predetermined threshold, the object identification information of the object shown in the specified outer region is identified using a second learning model or image matching process that has been machine-learned using image information of the specified outer region and second annotation data that associates image data of the object with object identification information;
If the reliability of the attribute corresponding to the specified outline region is equal to or less than a predetermined threshold, the object captured in the specified outline region is identified as an unknown object.
An information processing program characterized by: