JP7637563B2 - Recognition processing method and recognition processing device - Google Patents

Description

The present invention relates to a recognition processing method and a recognition processing device, and in particular to a technique for determining parameters in the recognition processing of a wide variety of objects.

A picking robot is an intelligent robot that has a robot arm with a gripper at the end and sorts items and palletizes/depalletizes cases based on information obtained by sensors. It is used in logistics warehouses, for example. The performance of a picking robot depends on the object recognition process used to analyze the information obtained by the sensors.

A technology that extracts features from image data acquired by a sensor and estimates the position of the object to be recognized in the image data based on the features is widely used as an object recognition process. In relation to the design of these features, there are generally several adjustable parameters that affect the recognition performance. Appropriate parameter values for a group of objects to be recognized can be calculated automatically by a computer if a model of the group of objects to be recognized and image data (hereinafter referred to as teaching data) in which the object is photographed and the object position is known for each object are prepared. However, the model and teaching data must be created and prepared manually. In particular, a large number of teaching data are required for each object to ensure the robustness of the calculated parameter values. Therefore, images must be taken of each object and object position information must be added, which requires a lot of manpower and labor costs. For example, in the case of a logistics warehouse, the number of objects to be recognized that require parameter determination is enormous, so it is necessary to perform parameter setting work efficiently.

In order to efficiently perform parameter setting work for many types of objects, one idea is to classify the objects into several classes and set appropriate parameter values for each class. For example, Patent Document 1 discloses a parameter setting method that identifies the class into which the object to be recognized is classified, and determines the parameter values corresponding to that class as appropriate parameter values for the object to be recognized, enabling highly accurate recognition.

JP 2020-68008 A

The parameter settings described in Patent Document 1 assume highly abstract patterns such as people and automobiles. Therefore, it is relatively easy to determine what patterns to prepare and how to classify the recognition target. However, if there is a large amount of diversity within the same pattern, it becomes difficult to achieve high recognition performance with the same parameter values. For example, in the case of cardboard boxes handled in a logistics warehouse, the sizes and patterns of the boxes vary widely, so it is necessary to further subdivide the patterns. However, as the pattern subdivision progresses, appropriate pattern classification becomes detached from human senses, and it becomes non-obvious what patterns to prepare and how to classify the items. In particular, when determining appropriate parameter values for a new item, in order to utilize pattern information, it is important to appropriately determine which pattern the item will be classified into.

The object of the present invention is to make it possible to determine parameters for object recognition processing for many types of objects using a small amount of teaching data.

A preferred example of the recognition processing method according to the present invention is a recognition processing method in which a computer executes an object recognition process, comprising the steps of:
storing a plurality of patterns including a plurality of registered articles, each pattern having a representative parameter value, in a storage unit;
receiving new item information having at least a model of the new item and a plurality of scene images;
a recognition processing step of performing a recognition process on a plurality of scene images of the novel article using the representative parameter values of each of the plurality of patterns;
selecting the pattern corresponding to the representative parameter value determined to be the best as a result of the recognition process as the most similar pattern to the new article;
and if the result of the recognition processing of the new article using the representative parameter value of the selected most similar pattern is good, determining the representative parameter value as a recognizable parameter value of the new article.

A preferred embodiment of the recognition processing device according to the present invention includes a storage unit that stores a plurality of patterns including a plurality of registered articles, each of which has a representative parameter value;
A receiving unit that receives at least a model of a new item and information on the new item having a plurality of scene images;
a recognition processing unit that performs a recognition process for a plurality of scene images of the novel article by using the representative parameter values of each of the plurality of patterns;
a determination unit that selects the pattern corresponding to the representative parameter value that is determined to be the best as a result of the recognition processing by the recognition processing unit as the most similar pattern of the new article;
and an output unit that outputs the representative parameter value as a recognizable parameter value of the new article if the result of the recognition process of the new article using the representative parameter value of the selected most similar pattern is good.
The present invention can also be understood as a program for causing a computer to execute the above recognition processing method.

The present invention makes it possible to determine parameters for object recognition processing for many types of objects using a small amount of teaching data.

FIG. 2 is a diagram showing an overview of an article recognition process according to the first embodiment. FIG. 13 is a diagram illustrating an example of parameter calculation. FIG. 13 is a diagram showing an example of pattern classification of a group of items. FIG. 2 is a diagram showing functional blocks of the item recognition processing apparatus. FIG. 11 is a diagram showing a processing operation of parameter determination in the article recognition processing device. 11A and 11B are diagrams illustrating an example of a recognition process using representative parameters. 11A and 11B are diagrams illustrating an example of determining a result of a recognition process using a representative parameter. FIG. 13 is a diagram showing an example of parameter determination based on a representative parameter value of a most similar pattern. 11A and 11B are diagrams illustrating an example of a method for adding item position information to a scene image. FIG. 13 illustrates an example of joint optimization. FIG. 13 is a diagram illustrating an example of parameter determination based on the result of joint optimization. FIG. 13 is a diagram illustrating an example of parameter determination based on a result of individual optimization. FIG. 2 is a diagram illustrating an example of a hardware configuration of an item recognition processing apparatus. FIG. 11 is a diagram illustrating a processing operation of parameter determination according to the second embodiment. 13A and 13B are diagrams illustrating an example of parameter determination and pattern updating based on a representative parameter value of a most similar pattern according to the second embodiment. FIG. 13 is a diagram showing an example of parameter determination and pattern update based on the result of joint optimization according to the second embodiment. FIG. 11 is a diagram showing an example of parameter determination and pattern update based on the results of individual optimization according to the second embodiment. FIG. 11 is a diagram illustrating a processing operation of parameter determination according to the third embodiment. FIG. 11 is a perspective view showing an application example of an article recognition processing apparatus.

Below, preferred embodiments of the present invention will be described with reference to the drawings. In describing the embodiments, the same parts in multiple drawings will be given the same reference numerals, and duplicate descriptions may be omitted.

The process of determining parameters in the item recognition processing device according to the first embodiment is described below.

FIG. 1 shows an overview of an article recognition process realized by computer processing.
The computer executes the recognition process using a model of the object to be recognized (hereinafter simply referred to as the object), a scene image including the object captured by a photographing means such as a camera, and parameter values of the recognition process as input data, and outputs estimated position information of the object in the scene image as output data. The processing procedure of the recognition process and details of each data depend on the recognition method used. The object model is data that characterizes the object, and includes CAD data of the object and feature amounts extracted from the image of the object. The scene image is an image having a format such as an RGB image or a depth image, or a combination thereof. A scene image generally includes one or more object objects, but there is no problem in the recognition process even if the object object is not included.

The parameter value θ for the recognition process is a value that determines the performance of the recognition process, such as the recognition rate and speed, and is generally expressed as a multidimensional vector quantity. The estimated position information that is output may be a set of pixel values of the image if you are interested in the position of the item on a two-dimensional image, or a set of coordinate values in three-dimensional space if you are interested in the position and orientation of the item in three-dimensional space. Although these are numerical data, they may also be expressed on a scene image, as shown in Figure 1.

If the scene image is provided with item position information, it is possible to determine whether the recognition result is a success or failure by comparing the item position information with the output estimated position information. Evaluation indices include IoU (Intersection Over Union) if you are interested in the item position on a two-dimensional image, and mean squared error if you are interested in the item position and orientation in three-dimensional space, and the value of the evaluation index is compared with a threshold value.

On the other hand, if the scene image does not have item position information, and the estimated position information can be displayed on a display device so that it is visible to humans, the operator can judge whether it has been successful by looking at the display. In this case, since the item position information in the scene image is not required, manpower and operating costs can be saved.

Here, we will explain one example of a recognition method. To obtain the three-dimensional position and orientation of a target object contained in a scene image, a set of an RGB image and a depth image is used as the scene image. To create a model of the target object, first, SIFT feature points and SIFT feature quantities are calculated from two-dimensional images obtained by photographing the target object from various angles. SIFT (Scale-Invariant Feature Transform) is a type of image feature quantity called a local feature, and SIFT feature points are expressed as pixel positions in two-dimensional images, and SIFT feature quantities are expressed as real vector quantities. A CAD model of the target object is prepared, and the SIFT feature quantities are linked to data points on the CAD model that correspond to the obtained SIFT feature points, and these are used as the model of the target object.

When the computer executes the recognition process, SIFT feature points and SIFT feature quantities are calculated from the RGB scene image, and three-dimensional coordinate values corresponding to those SIFT feature points are obtained from the depth scene image. Then, the correspondence between the three-dimensional feature points of the model and the scene image is found based on the feature quantity values, and a transformation matrix connecting the two is calculated from that correspondence. This transformation matrix corresponds to the estimated position information described above. In this recognition method, parameters related to the calculation of SIFT feature points and feature quantities, parameters related to finding a transformation matrix from the correspondence of three-dimensional feature points, and the like become parameters of the recognition process. Note that the processing procedure and effects of this embodiment are not limited to the above recognition method.

FIG. 2 shows an example of calculating appropriate parameter values for the recognition process.
The computer executes an optimization calculation based on input data including a model of the target item, multiple scene images including the target item, and item position information on the scene images, and calculates parameter values for the recognition process as output data. The optimization calculation is executed according to a processing procedure of an optimization method using parameter values that maximize an evaluation function that returns a value according to the parameter value. This makes it possible to obtain parameter values according to the purpose through the design of the evaluation function. For example, if the evaluation function is the recognition rate for multiple input scene images, the obtained parameter values are expected to maximize the recognition rate for the multiple input scene images. Furthermore, by adding a term that depends on the recognition processing speed, it is possible to calculate parameter values that enable the fastest possible recognition processing.

In the following explanation, we will assume an evaluation function that aims to obtain parameter values that will maximize the recognition rate for multiple input scene images. In this case, it should be noted that, in general, the greater the diversity of the input scene images, the wider the range of applicability of the obtained parameter values (the higher the robustness). In other words, to obtain robust parameter values, a large number of scene images (training data) with item position information attached are required, resulting in significant manpower and labor costs. As will be described later, this embodiment includes a processing procedure that makes it possible to determine highly robust parameters even with a small amount of training data.

Here, an example of an optimization method will be described. Examples of optimization methods include grid search and Bayesian optimization. Grid search, for example, divides the parameter space into grids of an appropriate size, calculates the evaluation function value at the parameter value corresponding to each grid point, and selects the maximum parameter value. Bayesian optimization searches for parameter values that are expected to produce a larger evaluation function value based on already calculated parameter values and the evaluation function values under those values. In optimizing parameters for recognition processing, recognition processing is often performed using certain parameter values to calculate the evaluation function value, and the calculation cost is relatively high. For this reason, an optimization method that is as efficient as possible is preferable. Note that the processing procedure and effects of this embodiment are not limited to the optimization method described above.

In the example of Figure 2, the input data shows a single type of target item, but two or more types of items may be used. In the example described below, the above process is used to find parameter values that enable simultaneous recognition of groups of different types of items.

Figure 3 shows an example of pattern classification of a group of items. In Figure 3, the horizontal axis represents each item pattern in a plurality of groups of items, and the vertical axis represents the parameter value. The vertical bar above each item represents the range of parameter values that realizes the required recognition performance for each item. For simplicity, one-dimensional parameters are assumed here, but two-dimensional or higher parameters can be considered in the same way. Please note that the items shown are examples for easy understanding of the explanation, and do not accurately reflect the actual ones. The parameter values that satisfy the required recognition performance change depending on the required recognition performance, but generally have a finite range. Therefore, there are cases where the required recognition performance can be achieved with the same parameter values even for different items. In this embodiment, focusing on this point, a group of items in which the same parameter values that can achieve the required recognition performance can be found is considered as one pattern. In the example of Figure 3, five items are classified into two pattern groups, pattern A consisting of three items and pattern B consisting of two items, by pattern classification. Here, the same parameter value found in each pattern is called a representative parameter value. This embodiment proposes a specific use of this pattern for determining appropriate parameter values for items.

To go further with regard to the above pattern classification, the pattern classification in this embodiment does not generally match human senses. In other words, it can be said that this pattern classification is a convenient pattern classification for the purpose of parameter determination, based on the criterion that groups of objects for which there are parameter values that simultaneously achieve the required recognition performance are classified as the same pattern. Furthermore, due to this clear criterion, as will be described later, it is possible to appropriately determine which pattern an object for which new parameters are to be set is classified into. Next, this pattern classification is not unique. It will become clear later that it is important to keep the number of patterns as small as possible, but the fact that it is not unique does not eliminate the effect of this embodiment. Also, even if the pattern classification is the same, there is a degree of freedom in the selection of representative parameter values. This also does not eliminate the effect of this embodiment.

FIG. 4 shows functional blocks of the article recognition processing apparatus.
The reception unit 401 receives input of information on new items (e.g., model information and scene images). The reception unit may also be called an input unit. The recognition processing unit 402 performs item recognition processing on the information input by the reception unit 401. The target item, scene image, and parameter value change depending on the transition source. The representative parameter value evaluation unit 403 evaluates the performance of the representative parameter value for the new item. For each representative parameter value, recognition processing is executed and a recognition rate is calculated based on the result of the recognition success/failure signal reception unit.

The most similar pattern determination unit 404 compares the recognition rates between the representative parameter values corresponding to the multiple patterns, and determines the pattern corresponding to the representative parameter value with the highest recognition rate as the most similar pattern. If the recognition rate is equal to or greater than the required value, it outputs it to the optimal parameter value output unit 409, and if the recognition rate is less than the required value, it transitions to the annotation signal reception unit 406.

The recognition success/failure signal receiving unit 405 receives a signal indicating whether the recognition result by the recognition processing unit 402 is successful or unsuccessful. In one example, an image of the recognition result is displayed on a display device, and an operator looks at the image to determine whether the recognition result is successful or unsuccessful, and the result of the determination is received from the input unit.

The annotation signal receiving unit 406 receives an annotation signal for a part of the failure scene of the new item with the representative parameter value of the most similar pattern. Then, when the result of the simultaneous optimization process by the optimization unit 407 is less than the required value and transitions from the recognition performance evaluation unit 408, the correct scene of the new item is also included as a target. If there is a failure scene of the new item to which correct answer information has not yet been assigned at this stage, the annotation signal receiving unit 406 can receive an annotation signal for that part. Here, the inventors mean that simultaneous optimization means, when the recognition rate of the new item does not satisfy the required performance, selecting the most similar pattern for the new item, adding the new item to that most similar pattern, and replacing (updating) the representative parameters (details will be described later).

The optimization unit 407 performs simultaneous optimization using the scene image of the new item to which the correct answer information has been assigned by the annotation signal receiving unit 406 and information on the most similar pattern. Alternatively, it performs individual optimization based on the information on the new item. The calculated parameter values are evaluated by the recognition performance evaluation unit 408. Here, the inventors mean that individual optimization means creating a new pattern that satisfies the required performance based on the information on the new item (details will be described later).

The recognition performance evaluation unit 408 evaluates the recognition rate for the new item of the parameter values calculated by the optimization unit 407 and checks whether it satisfies the required value. If the result of the joint optimization does not satisfy the required value, the unit returns to the optimization unit 407 and performs individual optimization.

The output unit 409 outputs the calculated optimal parameter value of the new object, and displays it, for example, on a display device. If the value is less than the required value, it is determined that recognition is impossible, and the process returns to the optimization unit 407. The pattern information DB 410 is a storage unit that stores information on registered object groups (e.g., model information and scene images) and pattern information extracted from them. The hardware configuration of the object recognition processing device will be described later with reference to FIG. 13.

Figure 5 is a flowchart of the parameter determination processing operation in the item recognition processing device. The parameters of a new item are determined according to the processing procedure shown in Figure 5. In this embodiment, to execute the item recognition process, input information is required that includes at least a model of the new item and a scene image. In the following explanation, it is assumed that parameters of one or more items have been determined, and that item pattern classification is performed based on that information.

First, the recognition processing unit 402 uses the representative parameter values of multiple patterns to recognize the new item in the scene image of the new item for the input information received by the reception unit 401 (S05). Based on the recognition result of the recognition processing unit 402 and in response to a signal from the recognition success/failure signal reception unit 405, the representative parameter evaluation unit 403 judges the recognition result for each representative parameter value and selects the representative parameter value that is deemed to be the best (S10). The pattern corresponding to the representative parameter value is called the most similar pattern.

The processing of steps S05-S10 will now be described in detail with reference to FIG. 6. Assume that the recognition processing of the new item is performed for each scene image of the new item using each representative parameter value of multiple patterns A, B, etc., and output data of estimated position information as shown in FIG. 6 is obtained. For ease of understanding, each estimated position information is shown on the corresponding scene image. Note that the recognition processing is performed for all scene images of the prepared new item, but the recognition processing may be limited to only a portion as necessary. The recognition is judged to be successful or unsuccessful based on each estimated position information, and an evaluation of the recognition performance of each representative parameter value (i.e., the recognition rate) is obtained. The evaluations for each pattern are compared and the one with the best evaluation is selected as the most similar pattern. In the example of FIG. 6, pattern A with a recognition rate of 0.95 is selected as the most similar pattern.

Here, if item position information is added to the scene image of the new item, the representative parameter evaluation unit 403 can automatically determine whether recognition has been successful or not using an appropriately set threshold value. However, at this stage, item position information is not necessarily required for the scene image of the new item. Therefore, in this embodiment, for example, a determination such as that shown in FIG. 7 is adopted. That is, each piece of estimated position information is expressed on the corresponding scene image and displayed on the display device. The operator looks at the display and determines whether recognition has been successful or not, and inputs the result of the determination (success or failure) from the input unit.

For scene images of new items for which the recognition result is judged to be successful, the output estimated position information is saved in the memory unit and can be used thereafter as the item position information for that scene image. This process makes it possible to automatically judge the result of the recognition of new items for that scene image. Note that there is no guarantee that the item position information at this point is optimal, so the following process is important.

Returning to FIG. 5, the most similar pattern determination unit 404 then determines whether the recognition results using the representative parameter values of the most similar pattern determined in step S10 are good (S20). This can be determined, for example, by comparing with a preset threshold value.

If it is judged to be good in step S20 (S20: YES), the parameter values capable of recognizing the new item are determined as the representative parameter values of the most similar pattern and output to the output unit 409 (S21). In this case, it is not necessary to calculate appropriate parameter values for the new item using the process shown in FIG. 2, and the values can be determined without adding item position information to the scene image of the new item. FIG. 8 shows an example of this process, in which pattern A is selected as the most similar pattern to the new item. As shown in FIG. 8, the representative parameter values of pattern A are values that are included in both the parameter range that satisfies the required recognition performance of the item group of pattern A and the parameter range that satisfies the required recognition performance of the new item.

At this stage, even if the subsequent processing in Figure 5 is not performed, the effect is that parameters can be determined with a small amount of teaching data. Subsequent processing can further enhance this effect.

If it is determined in step S20 that the result is not satisfactory (S20: NO), the annotation signal receiving unit 406 assigns item position information to a portion of the scene image of the new item that was determined to have failed to be recognized in the recognition process using the representative parameter value of the most similar pattern (S30). For example, FIG. 9 shows an example of assigning item position information to a scene image. As shown in FIG. 9, the scene image is displayed on a display device, and the operator can view the display and input appropriate item position information from the input unit. For example, if considering a bounding box on the scene image, it is only necessary to select four points on the scene image.

Next, the optimization unit 407 calculates parameters using the scene image of the new item and the most similar pattern (S40). For example, as shown in FIG. 10, in addition to the scene image to which the model of the new item and the item position information are attached, information on the most similar pattern is used as input data. Specifically, the information on the most similar pattern includes the model and scene image of the item classified into the most similar pattern, the item position information attached thereto, intermediate data obtained when the parameters of the item are determined, and the representative parameter value of the most similar pattern. At this point, since there are generally few scene images to which the item position information of the new item is attached that can be used for parameter calculation, information on the most similar pattern with the closest properties is used to compensate for this. As a result, even if there is little teaching data for the new item, it is expected that the calculated parameter values will be such that the new item can be sufficiently recognized. The higher the similarity between the new item and the most similar pattern (for example, the higher the recognition rate of the new item with the representative parameter value of the most similar pattern obtained in step S10), the higher the possibility is considered to be. Hereinafter, this parameter calculation is referred to as simultaneous optimization.

Here, some concrete examples of simultaneous optimization are described. The first example is an example of mixing teaching data. In the optimization process assumed in this example, a parameter value that will increase the recognition performance of the input scene image as much as possible is searched for. The teaching data of the item classified as the most similar pattern is considered to contain information for calculating the representative parameter value of the most similar pattern. On the other hand, if appropriate item position information is given, the scene image of the new item whose recognition result using the representative parameter value is judged to be successful is considered to contain at least part of the information for calculating the representative parameter value. In other words, even if the teaching data of the item classified as the most similar pattern is used instead of the scene images of those new items to which appropriate item position information is not given, it is expected that the same effect will be obtained in parameter calculation. The item position information is given to a part of the scene image of the new item whose recognition result using the representative parameter value of the most similar pattern is judged to be a failure in the process of step S30. Therefore, by using the item position information and the teaching data of the item classified as the most similar pattern as input data for parameter calculation, a parameter value that sufficiently increases the recognition performance of the new item can be obtained.

The second example of simultaneous optimization is an example in which the range of search in the parameter space is limited to the vicinity of the representative parameter value of the most similar pattern. In this case, the input teaching data is a scene image of a new item whose recognition result using the representative parameter value of the most similar pattern is judged to be a failure. The scene image may be given item position information in the process of step S30. It is expected that the parameter values in the vicinity of the representative parameter value of the most similar pattern can correctly recognize the scene image of a new item whose recognition result using the representative parameter value of the most similar pattern is judged to be a success, similar to the representative parameter value of the most similar pattern. Therefore, if a parameter value that can correctly recognize the input teaching data is found within the range of the scene image judged to be successful, it is a parameter value that sufficiently improves the recognition performance of the new item.
In this way, the search range can be restricted by simultaneous optimization, and the computational costs can be reduced.

Next, the recognition performance evaluation unit 408 judges whether the calculated parameter values are sufficient to recognize the new item (S50). If the judgment is to be made while also taking into consideration the recognition of scene images of new items to which no item position information is attached, a method such as that shown in FIG. 7 can be used.

If it is determined in step S50 that the item is recognizable (S50: YES), the calculated parameter value is determined as the recognizable parameter value of the new item (S51). Even if the representative parameter value cannot be used as is, by utilizing the information of the pattern that is most useful for calculating the appropriate parameter value, it is possible to determine the appropriate parameter value even if the item position information manually added to the scene image of the new item is small. FIG. 11 shows an example of this process, in which pattern A is selected as the most similar pattern of the new item. As shown in FIG. 11, the representative parameter value of pattern A is not included in the parameter range that satisfies the required recognition performance of the new item, but is closer to the range of the representative parameter value of pattern A than the representative parameter value of pattern B. Then, the parameter value that allows the new item to be recognized by simultaneous optimization is found relatively close to the representative parameter value of pattern A. It is not necessary that the parameter value at this stage allows the item group classified into pattern A to be recognized.

On the other hand, if it is determined in step S50 that the item is not recognizable (S50: NO), item position information is assigned to some of the scene images of the new item that do not have item position information assigned (S60). This can be achieved, for example, by using the method shown in FIG. 9.

Next, parameter calculation is performed (S70). Here, a model of the new item and a scene image of the new item with item position information are sufficient as input data. At this point, there is more teaching data for the new item available than at the time of processing in step S40, and it is highly likely that parameter values that will sufficiently improve the recognition performance of the new item will be calculated. Here, this optimization is referred to as individual optimization to distinguish it from simultaneous optimization.

Then, the recognition performance evaluation unit 408 judges whether the calculated parameter values are sufficient to recognize the new item (S80). Here, if the judgment is made taking into consideration the recognition of scene images of new items that do not have item position information, for example, a method such as that shown in FIG. 7 can be used.

If it is determined in step S80 that the new item is recognizable (S80: YES), the calculated parameter values are determined as the recognizable parameter values of the new item (S81). Figure 12 shows an example of this process, in which pattern A is selected as the most similar pattern of the new item. However, since the performance of the parameter values calculated in the joint optimization is not good, the parameter values calculated in the individual optimization of the new item are determined as the recognizable parameter values of the new item.

On the other hand, if it is determined in step S80 that the new item cannot be recognized (S80: NO), the currently assumed recognition method will not be able to achieve sufficient performance for the new item, and the item will be excluded from the parameter setting targets, for example, and the process will end.

As described above, the parameter determination process of this embodiment selects a pattern from the group of registered items that is expected to be most useful in determining appropriate parameter values for a new item, and by utilizing this information, it becomes possible to determine appropriate parameters even if the amount of teaching data for the new item is small.

<Hardware Configuration>
An example of the hardware configuration of the item recognition processing device will be described with reference to Fig. 13. The item recognition processing device 1 is a computer configured by a processor 101, a memory 102, an auxiliary storage device 103, an input device 104, an output device 105, and a communication IF (Interface) 106, all of which are connected to an internal bus 107.

The processor 101 executes a program stored in the memory 102 to realize an appropriate parameter calculation process. In this embodiment, a program that executes various functions related to the parameter calculation process (see FIG. 4) is assumed, but various other programs may also be executed. Input data and output data for executing the program are mainly stored in the memory 102. The memory 102 may be composed of, for example, a non-volatile storage element (e.g., a ROM (Read Only Memory)) for storing a program that does not need to be changed, and a volatile storage element (e.g., a RAM (Random Access Memory)) for temporarily storing the program to be executed and data used when the program is executed. The auxiliary storage device 103 includes a non-volatile large-capacity storage device such as a magnetic storage device (HDD: Hard Disk Drive), and stores the program executed by the processor 101 and data used when the program is executed (data including DB, etc.). The program for the parameter calculation process according to this embodiment is stored in the auxiliary storage device 103 in advance, read out from there, loaded into the memory 102, and executed by the processor 101.

The input device 104 is an input unit such as a keyboard or mouse that accepts input from an operator, and enables program execution and input data specification. The output device 105 is a device that visually outputs the results of program execution, such as a display device that displays information or a printer that prints information. In this embodiment, the display device displays information in a format that can be recognized by the operator, thereby realizing the processing shown in Figures 8 and 10. The communication IF 106 is a network interface device that connects the recognition processing device 1 to other devices and controls communication.

This hardware configuration may be configured as a single computer, or one or more of the components, processor 101, memory 102, auxiliary storage device 103, input device 104, output device 105, and communication IF 106, may be configured as one or more computers connected via a network.

In Example 2, the pattern is updated at the same time as the parameters of the object for which parameters are determined in Example 1. In Example 1, it is assumed that there is already available pattern classification information when a new object is registered, but in Example 2, a specific method for constructing the appropriate pattern classification information is presented.

FIG. 14 illustrates an example of a parameter determination process according to the second embodiment.
5 (parameter determination processing procedure) according to the first embodiment, steps S500, S210, S510, and S810 are different. The processing of these steps enables appropriate pattern updating accompanying registration of a new item, in addition to parameter determination of the new item. In the following, the description of the same processing as in FIG. 5 will be omitted, and the pattern updating processing operation and its action and effect according to the characteristic processing steps in FIG. 14 will be described.

In step S210, not only are representative parameter values of the most similar pattern selected as parameter values with which the new item can be recognized, but the pattern into which the new item is classified is designated as the most similar pattern. An example of this process is shown in FIG. 15. Because pattern A is selected as the most similar pattern for the new item, the representative parameter values of pattern A are selected as parameter values with which the new item can be recognized, and the new item is added to pattern A.

Next, in step S500, it is determined whether the parameter values calculated by the simultaneous optimization in step S40 are capable of simultaneously recognizing a group of objects classified into the most similar pattern in addition to a new object. Here, if the determination is made while taking into consideration the recognition of scene images to which object position information is not attached, this can be realized, for example, by the method shown in FIG. 7.

Step S510 is performed when it is determined that the parameter values calculated in step S500 can simultaneously recognize not only the new item but also the group of items classified as the most similar pattern. Not only are the parameter values determined as the recognizable parameter values of the new item, but the new item is classified as the most similar pattern and the representative parameter value of the most similar pattern is replaced with the calculated parameter value. An example of this process is shown in FIG. 16. Pattern A is selected as the most similar pattern of the new item, but it is determined that the new item cannot be recognized well using the representative parameter value, so simultaneous optimization is performed and a new parameter value is calculated. The value is included in both the parameter range that satisfies the required recognition performance of the group of items in pattern A and the parameter range that satisfies the required recognition performance of the new item, and can be used as the new representative parameter value of pattern A. The new item is then classified into pattern A.

The determination in step S500 shows that this pattern update meets the criteria for pattern classification in Example 2. Furthermore, this pattern update is able to incorporate new items without changing the number of patterns. This pattern update is preferable because an increase in the number of patterns increases the amount of processing in steps S20 and S500. Furthermore, the representative parameter values can also be updated, allowing for flexible pattern updating. This has the advantage that the representative parameter values can be calculated by simultaneous optimization even when the amount of teaching data for the new item is small.

Finally, in step S810, not only are the parameter values calculated in the individual optimization in step S70 determined as parameter values capable of recognizing the new item, but a new pattern is generated in which only the new item is a belonging item and the representative parameter values are calculated. An example of this process is shown in FIG. 17. Pattern A is selected as the most similar pattern of the new item, but since it is deemed that the recognition of the new item using the representative parameter value is not good, simultaneous optimization is performed and new parameter values are calculated. The parameter values calculated in the simultaneous optimization are sufficient to recognize the new item. However, since it is not possible to sufficiently recognize all items classified into pattern A, individual optimization is performed in Example 2. Since the parameter values calculated in the individual optimization are sufficient to recognize the new item, the parameter values are determined as parameter values capable of recognizing the new item. Furthermore, a new pattern C is generated, the new item is classified into it, and the representative parameter values are those calculated in the individual optimization.

By updating the patterns in step S810, new items that are determined not to be classified into any patterns will be classified into the newly generated pattern. In particular, it is possible to construct a pattern even in a situation where there are no registered items.

Now, the function for performing the above pattern update process will be described. The functions that differ from those in the first embodiment will be described with reference to FIG. 4. The recognition performance evaluation unit 408 evaluates the recognition rate for the new item of the parameter values calculated by the optimization unit, and checks whether it satisfies the required value. In the case of simultaneous optimization, the recognition rate is evaluated not only for the new item, but also for multiple items (item groups) that belong to the most similar pattern. If the result of simultaneous optimization does not satisfy the required value, the process returns to the annotation signal reception unit 406.

The pattern information DB 410 stores information on registered item groups (model information and scene images) and pattern information extracted from them. In the second embodiment, this information is appropriately changed when outputting optimal parameters for a new item.

As described above, according to the parameter determination and update of the second embodiment, by selecting a pattern from the group of registered items that is expected to be most useful in determining appropriate parameter values for a new item, and utilizing that information, it is possible to determine appropriate parameters for the new item even if the amount of teaching data is small. Furthermore, it is possible to update the pattern so that it becomes a useful pattern for the item for which parameters are to be determined subsequently.

Example 3 presents parameter determination that can repeat the simultaneous optimization process in the parameter determination of Example 1. FIG. 18 shows an example of a parameter determination process procedure according to Example 3. In comparison with Example 1, step S52 is newly added to the process procedure of FIG. 4, which enables loop processing. Steps other than step S52 and the functions that realize them are substantially the same as those of Example 1, so their description will be omitted.

In this embodiment, if it is determined in step S50 that the parameter values calculated by the simultaneous optimization in step S40 are not sufficient to recognize the new item (S50: NO), it is determined whether item position information has been added to all of the scene images of the new item that were determined to have failed to be recognized using the representative parameter values of the most similar pattern in step S20 (S52). If the result of this determination is that item position information has been added to all of the images (S52: YES), the process proceeds to step S60, and thereafter the same processing procedure as in embodiment 1 is followed.

On the other hand, if the result of the above judgment is that there is no item position information assigned (S52: NO), the process returns to step S30, and item position information can be assigned to the scene images to which item position information has not been assigned. That is, simultaneous optimization is performed in step S40, but the difference from the previous simultaneous optimization is that there are more scene images to which item position information of new items has been assigned this time. Therefore, the calculated parameter values are more likely to be able to fully recognize the new item in step S50. If the new item cannot be fully recognized, and there are still scene images to which item position information can be assigned, the process returns to step S30 again in step S52, and the above-mentioned process can be repeated. The possibility of exiting the repeated process is when it is determined in the process in which it is determined in step S50 that parameter values capable of fully recognizing the new item have been calculated (step S51), when it is determined that item position information has been assigned to all failure scene images, or when it is determined in step S51 that there is no need to perform further simultaneous optimization.

In this way, according to the third embodiment, compared to the first embodiment, it is possible to gradually assign item position information to new items until the results of simultaneous optimization become favorable, which has the effect of obtaining parameter values that can adequately recognize new items with the lowest possible cost for creating teaching data.

Example 4 is an example in which multiple representative parameters (called a representative parameter value group) are considered as representative parameter values.

[Response to Example 1]
In the recognition process in the first embodiment (see FIG. 5), each pattern has one representative parameter value. However, it is possible to find multiple parameter values that make the recognition performance of the belonging article equal to or higher than the required performance, and therefore, in this embodiment, it is possible to assign multiple representative parameter values to each pattern.

For example, when selecting the most similar pattern, the recognition performance of the new item is evaluated for all representative parameter values for each pattern, and the pattern containing the representative parameter value with the best recognition performance is selected as the most similar pattern. As the number of parameter value evaluations increases, the computational costs and the costs required to determine the recognition results increase, but more granular judgments become possible. In this case, the representative parameter value used when determining the optimal parameter value for the new item can be the representative parameter value with the highest recognition rate that was focused on when selecting the most similar pattern among the multiple representative parameter values of the most similar patterns.

[Response to Example 2]
In the above embodiment 1, an example was described in which a plurality of representative parameter values for each pattern were considered in order to select the most similar pattern with finer granularity and determine the optimal parameter value with higher recognition performance. A similar example can be described in the embodiment 2. In this case, in addition to the changes in the embodiment 1, a change is required to the process related to updating the pattern information. For example, when changing the representative parameter value of the most similar pattern in step S510, a plurality of representative parameter values are changed, and when adding a new pattern in step S810, a plurality of representative parameter values are obtained. For the former, steps S40 and S500 are repeated to obtain the required number of representative parameter values, and the required number of representative parameter values are replaced with the existing representative parameter values of the most similar pattern. Similarly, for the latter, steps S70 and S80 are repeated until the required number of representative parameter values are obtained. Here, if at least one new representative parameter value has been obtained, it is expected that other values that can become the representative parameter value are near that value, so it is possible to limit the search range to the vicinity of that value and suppress the increase in the calculation cost to some extent.

With reference to FIG. 19, an example in which the item recognition processing device according to the above-mentioned Examples 1-4 is applied to item recognition in a picking robot will be described. In FIG. 19, a computer that realizes the item recognition processing device 1 has a mouse as an input device 104 and a display as an output device 105. A robot 1902 and an image sensor 1903 are connected to the communication IF 106 of the item recognition processing device 1 via a communication line 1901. The robot 1902 receives the recognition result by the item recognition processing device 1 and performs the desired operation. The image sensor 1903 captures an object 1931 that is the target of the operation of the robot 1902 and a work scene for the object. Note that the sensor and computer may be part of the robot.

For example, the flow of operations of the robot shown in FIG. 19 is as follows. First, the item recognition processing device 1 executes recognition processing on an image acquired by a sensor 1903 installed on or around the body of the robot 1902, and obtains the recognition result. Then, the computer analyzes the recognition result using an appropriate program to obtain information that induces appropriate robot operations. Based on that information, the robot 1902 is made to perform an operation. For example, the object 1931 to be recognized is grasped, and the grasped object is moved to a specified position. Also, although the work scene in FIG. 19 is depicted in a fixed state, the work scene can be changed as appropriate. In particular, it is possible to use situations that change from moment to moment, such as an actual production line.

When Example 1 is applied to this application example, the following effect is obtained. That is, the success or failure of the recognition result, which occurs in step S10 or the like, can be determined based on the movement of the robot 1902. As shown in FIG. 7, when the recognition result is displayed on a display device, it cannot be denied that the operator may determine that the recognition is successful even when the accuracy is insufficient to achieve the purpose, or that the recognition is unsuccessful even when the accuracy is sufficient. Therefore, according to this application example, a more appropriate judgment can be made by having the robot perform the actual intended movement.

1: Item recognition processing device 101: Processor 102: Memory 103: Auxiliary storage device 104: Input device 105: Output device 106: Communication IF 107: Internal bus 401: Reception unit 402: Recognition success/failure signal reception unit 403: Representative parameter value evaluation unit 404: Most similar pattern determination unit 405: Pattern information DB 406: Annotation signal reception unit 407: Optimization unit 408: Recognition performance evaluation unit 409: Output unit 410: Storage unit

Claims (12)

A recognition processing method in which a computer executes an article recognition process, comprising the steps of:
storing a plurality of patterns including a plurality of registered articles, each pattern having a representative parameter value, in a storage unit;
receiving new item information having at least a model of the new item and a plurality of scene images;
a recognition processing step of performing a recognition process on a plurality of scene images of the novel article using the representative parameter values of each of the plurality of patterns;
selecting the pattern corresponding to the representative parameter value determined to be the best as a result of the recognition process as the most similar pattern to the new article;
If a result of a recognition process for the new article using the representative parameter value of the selected most similar pattern is good, determining the representative parameter value as a recognizable parameter value of the new article;
The recognition processing method includes: If the result of the recognition process (re-recognition process) of the new article using the representative parameter value of the most similar pattern is not good,
selecting at least one of the plurality of scene images of the novel article that is determined to have failed as a result of the recognition process;
Adding item position information to the selected scene image;
calculating new parameter values based on the selected scene image, the item position information, the model of the new item, and the representative parameter values of the most similar pattern or information for calculating the representative parameter values of the most similar pattern;
performing a recognition process for the new article on the plurality of scene images of the new article using the model of the new article and the new parameter values;
If the result of the recognition process is good, determining the new parameter value as a recognizable parameter value of the new article;
The recognition processing method according to claim 1 Each of the plurality of patterns has, as pattern information, at least models of all the registered articles classified into the pattern and a plurality of scene images in which all the registered articles classified into the pattern are captured;
if a result of the recognition process using the representative parameter value of the most similar pattern is good, the model of the new article and the plurality of scene images in which the new article is captured are added to the pattern information of the most similar pattern, and the new article is classified into the most similar pattern;
if the result of the recognition process (re-recognition process) of the new item is good for the plurality of scene images of the new item using the model of the new item and the new parameter values, for each of all registered items classified into the most similar pattern, a recognition process of the registered item is performed for the plurality of scene images in which the registered item is photographed using the model of the registered item and the new parameter values;
if the results of the recognition process for all of the registered articles classified into the most similar pattern are good, add the model of the new article and the plurality of scene images in which the new article is photographed to the pattern information of the most similar pattern;
replacing the representative parameter value of the most similar pattern with the new parameter value, and classifying the new article into the most similar pattern;
The recognition processing method according to claim 2 . If the results of all of the recognition processes are not good,
selecting a portion of the plurality of scene images of the novel item;
Adding item position information to each of the selected scene images;
Calculating new parameter values based on a portion of the plurality of scene images of the new item to which item position information is assigned, the item position information, and the model of the new item;
performing a recognition process (re-recognition process) of the new article on the plurality of scene images of the new article using the model of the new article and the new parameter values;
If the result of the re-recognition process is good, the new parameter values are determined as recognizable parameter values of the new article;
retaining at least the model of the new article and the plurality of scene images in which the new article is captured as pattern information, creating a new pattern in which the new parameter value is a representative parameter value, and classifying the new article into the new pattern;
The recognition processing method according to claim 3. displaying, on a display device, information on an estimated position of the article relative to a scene image of the article together with the scene image;
inputting a result of the recognition processing based on the display through an input device;
The recognition processing method according to claim 1 . The target is to determine the parameters of the robot's object recognition process.
the robot executes a predetermined operation based on estimated position information of the object estimated by object recognition for a scene image of the object, the object recognition being executed using the recognition process, a model of the object, and arbitrary parameter values;
determining a result of the recognition process based on the motion;
The recognition processing method according to claim 1 . When the result of the re-recognition process is not good and the result of the recognition process using the representative parameter value of the most similar pattern is determined to be a failure, there is a scene image of the new article to which no article position information is assigned, and the process according to claim 2 is executed again.
The recognition processing method according to claim 2 . each of the plurality of patterns is a representative parameter value group having a plurality of parameter values that enable recognition of a plurality of the registered articles classified into the corresponding pattern;
performing a recognition process for the new article on the plurality of scene images of the new article by using the representative parameter value group of each of the plurality of patterns;
selecting the pattern corresponding to the representative parameter value group having the best parameter value as a result of the recognition process as the most similar pattern to the new article;
determining, when a result of the recognition process using the parameter values belonging to the representative parameter value group of the most similar pattern is good, the parameter values belonging to the representative parameter value group of the most similar pattern as recognizable parameter values of the novel article;
The recognition processing method according to claim 1 . A storage unit that stores a plurality of patterns including a plurality of registered articles, each of the patterns having a representative parameter value;
A receiving unit that receives at least a model of a new item and information on the new item having a plurality of scene images;
a recognition processing unit that performs a recognition process for a plurality of scene images of the novel article by using the representative parameter values of each of the plurality of patterns;
a determination unit that selects the pattern corresponding to the representative parameter value that is determined to be the best as a result of the recognition processing by the recognition processing unit as the most similar pattern of the new article;
an output unit that outputs the representative parameter value as a recognizable parameter value of the new article when a result of a recognition process of the new article using the representative parameter value of the selected most similar pattern is good;
A recognition processing device having the above configuration. If the result of the recognition process of the new article using the representative parameter value of the most similar pattern is not good,
selecting at least one of the plurality of scene images of the novel article that is determined to have failed as a result of the recognition process;
Adding item position information to the selected scene image;
the determination unit calculates new parameter values based on the selected scene image, the item position information, the model of the new item, and the representative parameter values of the most similar pattern or information for calculating the representative parameter values of the most similar pattern;
the recognition processing unit performs a recognition process of the new article on the plurality of scene images of the new article by using the model of the new article and the new parameter values;
When a result of the recognition process is good, the determination unit determines the new parameter value as a recognizable parameter value of the new article.
The recognition processing device according to claim 9 . Each of the plurality of patterns has, as pattern information, at least models of all the registered articles classified into the pattern and a plurality of scene images in which all the registered articles classified into the pattern are captured;
when a result of the recognition process using the representative parameter value of the most similar pattern is good, the determination unit adds the model of the new article and the plurality of scene images in which the new article is captured to the pattern information of the most similar pattern, and classifies the new article into the most similar pattern;
the recognition processing unit, when a result of a recognition process (re-recognition process) of the new item for the plurality of scene images of the new item using the model of the new item and the new parameter values is good, performs a recognition process of the registered item for each of all registered items classified into the most similar pattern, using the model of the registered item and the new parameter values for the plurality of scene images in which the registered item is photographed;
when a result of the recognition process of all the registered articles classified into the most similar pattern is good, the determination unit adds the model of the new article and the plurality of scene images in which the new article is photographed to the pattern information of the most similar pattern;
replacing the representative parameter value of the most similar pattern with the new parameter value, and classifying the new article into the most similar pattern;
The recognition processing device according to claim 10. If the results of all of the recognition processes are not good,
selecting a portion of the plurality of scene images of the novel item;
Adding item position information to each of the selected scene images;
The determination unit calculates new parameter values based on a part of the plurality of scene images of the new item to which item position information is assigned, the item position information, and the model of the new item;
the recognition processing unit executes a recognition process (re-recognition process) of the new article on the plurality of scene images of the new article using the model of the new article and the new parameter values;
The determination unit determines the new parameter values as recognizable parameter values of the new article when a result of the re-recognition process is good,
retaining at least the model of the new article and the plurality of scene images in which the new article is captured as pattern information, creating a new pattern in which the new parameter value is a representative parameter value, and classifying the new article into the new pattern;
The recognition processing device according to claim 11.

Images