JP6919276B2 - Image display program, image display device and image display method - Google Patents

Description

The present invention relates to an image display program, an image display device, and an image display method.

In a system for registering data in a database by associating tags, a tag candidate selection method for displaying tag candidates associated with the data when newly registering data in the database has been proposed (see, for example, Patent Document 1). ). In this type of method, the similarity between multiple registered data already registered in the database and the newly registered new data is calculated, and a tag associated with the registered data similar to the new data is newly added. Display as a candidate tag to be associated with data.

Further, in a system for calculating the similarity between data using a plurality of types of feature amounts representing data features, a method of weighting between the plurality of types of feature amounts has been proposed (see, for example, Patent Document 2). ..

Further, in the image dictionary generator that generates an image dictionary, the accuracy of the image dictionary is improved by dividing the meaning label given to the image into more detailed topics based on the feature amount of the trained image prepared in advance. A method has been proposed (see, for example, Patent Document 3).

JP-A-2007-272390 Japanese Unexamined Patent Publication No. 2004-13307 Japanese Unexamined Patent Publication No. 2014-115737

In the discrimination system that analyzes the captured image of the object to be inspected and discriminates between non-defective products and defective products, the feature amount extracted from the image is compared with a preset threshold value, and whether the product is non-defective based on the comparison result. Determine if it is defective. The threshold value is set based on, for example, a feature amount extracted from a plurality of images in which a non-defective product image in which a non-defective product is imaged or a defective product image in which a defective product is imaged is identified in advance.

For example, an operator such as an inspector who sets a threshold for classifying an object to be inspected into a non-defective product and a defective product visually observes a plurality of images of the object to be inspected and divides the image into a non-defective product image and a defective product image. Classify. Then, the operator executes labeling to give information (hereinafter, also referred to as a label) indicating the type of the image such as a non-defective product image or a defective product image to the image.

By increasing the number of images used to set the threshold, the reliability of the set threshold is improved, but the number of images to be labeled also increases. As the number of images to be labeled increases, the operator may inconsistent with past labeling results by the operator when labeling the images. That is, the image may be given the wrong label. If the threshold is set using multiple images, including images with the wrong label, the reliability of the threshold is reduced.

In the method of displaying the label candidates to be given to the image when newly registering the image in the database or the like, the label displayed as the label candidate to be given to the image is given to any registered image. It is difficult to confirm the label. Therefore, when the operator assigns a label to the image, the operator may cause a contradiction with the past labeling result.

In one aspect, the present invention aims to prevent the occurrence of inconsistencies with past labeling results when labeling images.

In one embodiment, the image display device includes a feature amount calculation unit, a first image identification unit, a second image identification unit, and a display unit. The feature amount calculation unit calculates a plurality of types of feature amounts from the input image. The first image identification unit is associated with the first type among the types by referring to a storage unit that stores information indicating the type of the image associated with the image and a plurality of types of feature quantities calculated from the image. Among the images, the first image most similar to the input image is specified. The second image identification unit is associated with the first type of a plurality of images stored in the storage unit in the coordinate space centered on each type of feature amount, and is associated with the input image and the first type. A second image having a predetermined positional relationship with the image of is specified. The display unit displays a screen including information indicating the first type, the first image, the second image, and the input image.

On one aspect, the present invention can prevent the occurrence of inconsistencies with past labeling results when labeling an image.

It is a figure which shows one Embodiment of an image display program, an image display device, and an image display method. It is a figure which shows an example of the operation of the image display device shown in FIG. It is a figure which shows another embodiment of an image display program, an image display device, and an image display method. It is a figure which shows an example of the feature amount database shown in FIG. It is a figure which shows an example of the operation of the similarity calculation part shown in FIG. It is a figure which shows an example of the operation of the 1st image specifying part and the 2nd image specifying part shown in FIG. It is a figure which shows an example of the operation of the display part shown in FIG. It is a figure which shows an example of the operation of the image display device shown in FIG. It is a figure which shows another embodiment of an image display program, an image display device, and an image display method. It is a figure which shows an example of the operation of the 2nd image specifying part shown in FIG. It is a figure which shows an example of the operation of the image display device shown in FIG. It is a figure which shows another embodiment of an image display program, an image display device, and an image display method. It is a figure which shows an example of the operation of the 2nd image specifying part shown in FIG. It is a figure which shows another embodiment of an image display program, an image display device, and an image display method. It is a figure which shows an example of the operation of the 3rd image identification part shown in FIG. It is a figure which shows an example of the operation of the display part shown in FIG. It is a figure which shows an example of the operation of the image display device shown in FIG.

Hereinafter, embodiments will be described with reference to the drawings. Hereinafter, the image itself and the image data for displaying the image are referred to as an image without particular distinction.

FIG. 1 shows an embodiment of an image display program, an image display device, and an image display method. The image display device 200 shown in FIG. 1 is based on the past labeling result when, for example, an operator such as an inspector assigns a label indicating the type of the image IMG (for example, a non-defective product, a defective product, etc.) to the image IMG. Execute image display processing to suppress the occurrence of contradictions. The label is an example of information indicating the type of image IMG. The image display device 200 may be realized only by hardware, or may be realized by controlling the hardware by software. In the example shown in FIG. 1, the image display device 200 is realized by the computer 10 executing the image display program DPRG.

For example, the computer 10 has a processor 20 such as a CPU (Central Processing Unit), a memory 30, a hard disk device 40, an input interface 50, an output interface 60, and a drive device 70. The processor 20, the memory 30, the hard disk device 40, the input interface 50, the output interface 60, and the drive device 70 are connected to the bus 80. The processor 20 executes a program stored in the memory 30, the hard disk device 40, or the like. The input interface 50 transfers the image IMG received from the image pickup device 90 such as a digital camera, the information received from the input device 100 such as a keyboard, and the like to the processor 20 and the like. Further, the output interface 60 outputs the execution result of the program by the processor 20 to the display device 110 such as a display. The drive device 70 can be equipped with the removable media 120, which is a recording medium that can be read by the computer 10, and reads and records the information recorded on the mounted removable media 120.

The image display device 200 is realized by the processor 20 executing the image display program DPRG stored in the memory 30. When the image display device 200 assigns a label to an input image IMG such as an image IMG captured by the image pickup device 90, the image display device 200 has already labeled the reference image IMGr to be displayed on the display device 110 together with the input image IMG. Select from. The labeled image IMG is stored, for example, in memory 30. Further, the memory 30 stores information indicating the type of the image IMG and a plurality of types of feature amounts representing the features of the image IMG in association with the image IMG. The memory 30 is an example of a storage unit. The labeled image IMG or the like may be stored in the hard disk device 40. In this case, the hard disk device 40 is also an example of the storage unit.

For example, the image display device 200 has a feature amount calculation unit 210, a first image identification unit 220, a second image identification unit 230, and a display unit 240.

The feature amount calculation unit 210 receives the input image IMGi from the image pickup apparatus 90 via the input interface 50, for example. Alternatively, the feature amount calculation unit 210 may receive the input image IMGi from the memory 30 or the like. Then, the feature amount calculation unit 210 extracts an object to be inspected from the input image IMGi by a background subtraction method or the like that extracts a different portion between the input image IMGi and the background image captured in advance, and obtains the features of the extracted object. Extract multiple types of features to be represented. The background image is an image in a state where there is no object to be inspected. In this way, the feature amount calculation unit 210 calculates a plurality of types of feature amounts from the input image IMGi.

The plurality of types of features extracted from the input image IMGi by the feature calculation unit 210 include, for example, the shape features described in "Mikio Takagi, Yoshihisa Shimoda: New Image Analysis Handbook, University of Tokyo Press, 2004." Texture features, color features, etc., which are predetermined. Shape features are area, circumference length, circularity, maximum length, width, etc., texture features are luminance histogram, edge density, simultaneous occurrence matrix statistic, Fourier transform power spectrum, etc., and color features are It is a color histogram and the like. The type of feature amount extracted from the input image IMGi is not limited to the above example. Further, the method of extracting an object from the input image IMGi is not limited to the background subtraction method.

The first image specifying unit 220 refers to a memory 30 that stores a label indicating the type of the image IMG and a plurality of types of feature quantities calculated from the image IMG in association with the already labeled image IMG, and stores the input image. Calculate the similarity between the IMGi and the already labeled image IMG. For example, the first image identification unit 220 compares the feature amount calculated from the input image IMGi with the feature amount of the already labeled image IMG, and the similarity between the input image IMGi and the already labeled image IMG. Is calculated. Then, the first image identification unit 220 selects one of the plurality of types of the image IMG as the first type, and among the image IMGs to which the label indicating the selected first type is attached, the input image IMGi is used. Identify the first image IMG with the most similar features.

In this way, the first image identification unit 220 identifies the first image IMG that is most similar to the input image IMGi among the image IMGs associated with the first type of the image IMG types. The identification of the first image IMG and the identification of the second image IMG, which will be described later, are executed for each type of image IMG.

The second image identification unit 230 is associated with the first type among the plurality of image IMGs stored in the memory 30, and is located at a predetermined position with the input image IMGi and the first image IMG in the feature space. Identify the second image IMG that is relevant. The feature space is a coordinate space centered on each type of feature amount calculated by the feature amount calculation unit 210. For example, when there are two types of feature quantities, size and brightness, the feature space is two-dimensional and has a horizontal axis indicating size and a vertical axis indicating brightness, as shown in parentheses in FIG. .. In this way, the feature space is determined by the type of feature quantity. The details of the operation of the second image specifying unit 230 will be described with reference to FIG. The first image IMG and the second image IMG are reference image IMGrs to be displayed on the display device 110 together with the input image IMGi. Hereinafter, the first image IMG and the second image IMG may use the same reference numerals as the reference image IMGr. Further, the first image IMG and the second image IMG may be referred to as a reference image IMGr.

The display unit 240 displays on the display device 110 a screen including information indicating the first type, the first image IMG, the second image IMG, and the input image IMGi. For example, the display unit 240 outputs the information PINF including the input image IMGi, the reference image IMGr, and the label of the reference image IMGr to the display device 110 via the output interface 60. As a result, the labels of the input image IMGi, the reference image IMGr, and the reference image IMGr are displayed on the display device 110.

The configuration of the image display device 200 and the computer 10 is not limited to the example shown in FIG. For example, the image display program DPRG may be stored in a removable media 120 such as a CD-ROM (Compact Disc Read only memory), a DVD (Digital Versatile Disc), or a USB (Universal Serial Bus) memory. In this case, the image display program DPRG stored in the removable media 120 is transferred from the removable media 120 to the memory 30 via the drive device 70. Further, the image display program DPRG may be transferred from the removable media 120 to the hard disk device 40 and then to the memory 30.

Alternatively, the computer 10 may download an application program such as the image display program DPRG via a communication device (not shown) connected to a network such as the Internet and store it in the memory 30 or the hard disk device 40. Further, the computer 10 may acquire an image IMG from the image pickup device 90 via a communication device connected to the network. Further, for example, in the computer 10, the drive device 70 may be omitted. Further, for example, the computer 10 may have an SSD (Solid State Drive) or the like instead of the hard disk device 40.

FIG. 2 shows an example of the operation of the image display device 200 shown in FIG. Note that FIG. 2 shows an example of the operation of the image display device 200 when there are two types of feature quantities, size and brightness. The operation shown in FIG. 2 is an aspect of the image display method executed by the computer 10. Further, the program for causing the computer 10 to execute the operation shown in FIG. 2 is one aspect of the image display program. The operation shown in FIG. 2 may be realized only by hardware. The series of processes from step S100 to step S700 is executed each time the image display device 200 receives the input image IMGi. Further, the graph shown in parentheses in FIG. 2 shows an outline of a process for specifying the first image IMG and the second image IMG (processes in steps S400 and S500). The horizontal axis of the graph shown in FIG. 2 indicates the size of the object, and the vertical axis indicates the brightness of the object. Further, the shaded quadrangle shown in FIG. 2 indicates an image IMGa that has already been given a label indicating type a (for example, a non-defective product), and the triangle indicates an image IMGa that has already been given a label indicating type b (for example, a defective product). The image IMGb is shown, and the star shape indicates the input image IMGi to be labeled. The number of types of image IMG and the type of feature amount are not limited to two. The images IMGa and IMGb are stored in, for example, the memory 30.

In step S100, the feature amount calculation unit 210 calculates the feature amount (size and brightness in the example shown in FIG. 2) of the input image IMGi as described with reference to FIG. As a result, the input image IMGi is arranged on the feature space having a horizontal axis indicating the size and a vertical axis indicating the brightness, as shown in the parentheses of FIG.

Next, in step S200, the first image identification unit 220 compares the feature amount of the input image IMGi with the feature amount of the other images IMGa and IMGb, and resembles the input image IMGi with the other images IMGa and IMGb. The degree is calculated for each of the other images IMGa and IMGb. The degree of similarity is expressed by the distance between the features. The vector V (Va1, Va2, Vaj) shown in FIG. 2 indicates the position of the target image IMG (IMGa1, IMGa2, IMGaj) when the position of the input image IMGi in the feature space is used as the starting point. Therefore, the distance between the feature amount of the input image IMGi and the feature amount of the target image IMG is represented by the magnitude of the vector V. For example, the distance between the feature amount of the input image IMGi and the feature amount of the image IMGa1 is represented by the magnitude of the vector Va1 (= | Va1 |).

Next, in step S300, the first image identification unit 220 sets one of the plurality of types of the image IMG as the first type. For example, the first image identification unit 220 sets one of the type a and the type b as the first type.

Next, in step S400, the first image identification unit 220 identifies the first image IMG that is most similar to the input image IMGi among the image IMGs associated with the first type. The similarity between the two image IMGs is higher as the distance between the features of the two image IMGs is shorter. Therefore, in the example shown in FIG. 2, when the first type set in step S300 is the type a, the first image specifying unit 220 is the image having the smallest vector Va size among the image IMGas of the type a. IMGa1 is identified as the first image IMG.

Next, in step S500, the second image identification unit 230 uses the second image IMG that has a predetermined positional relationship with the input image IMGi and the first image IMG among the image IMGs associated with the first type. To identify. In the example shown in FIG. 2, the predetermined positional relationship is represented by a vector V orthogonal to or substantially orthogonal to the vector V indicating the position of the first image IMG when the position of the input image IMGi in the feature space is used as the starting point. It is a positional relationship.

For example, the second image identification unit 230 identifies the image IMGa2 represented by the vector Va2 orthogonal to the vector Va1 whose start point is the input image IMGi and whose end point is the first image IMGa1 as the second image IMG. When a plurality of image IMGas represented by the vector Va orthogonal to the vector Va1 are extracted, the second image identification unit 230 selects the image IMGa at the position closest to the input image IMGa among the extracted plurality of image IMGas. Specific to the image IMG.

The predetermined positional relationship is not limited to the positional relationship indicated by the vector V orthogonal to or substantially orthogonal to the vector V indicating the position of the first image IMG. For example, the predetermined positional relationship may be such that the distance from the first image IMG on the feature space is equal to or greater than the predetermined first distance. In this case, the second image specifying unit 230 uses the image IMG at the position closest to the input image IMG among the image IMGs whose distance from the first image IMG in the feature space is equal to or greater than the predetermined first distance. Specific to the image IMG.

Among the image IMGs associated with the first type set in step S300 by the processing of steps S400 and S500, the reference image IMGr (the first image IMG and the first image IMG) to be displayed on the display device 110 together with the input image IMGi. 2 image IMG) is identified.

Next, in step S600, the image display device 200 determines whether or not all types of image IMGs have been searched. When all types of image IMGs have been searched, the operation of the image display device 200 proceeds to step S700.

On the other hand, if there is a type that has not been searched, the operation of the image display device 200 returns to step S300. For example, when the search for the reference image IMGr in the type b of the type a and the type b is not executed, the image display device 200 sets the type b as the first type in step S300. As a result, the processes of steps S400 and S500 are executed for the image IMGb of the type b, and the images IMGb1 and IMGb2 are specified as the reference image IMGr.

In step S700, the display unit 240 displays a screen including the first type (type a and type b), the first image IMGa1, IMGb1, the second image IMGa2, IMGb2, and the input image IMGi on the display device 110. .. The images IMGa1, IMGa2, IMGb1, and IMGb2 are displayed so that each type of image IMG can be identified. When the process of step S700 is completed, the operation of the image display device 200 for one input image IMGi is completed.

As described above, in the operation shown in FIG. 2, the image IMAGaj is similar to the input image IMGi as compared with the image IMGa2, but the positional relationship between the input image IMGi and the first image IMG does not satisfy a predetermined positional relationship. Therefore, it is not selected as the reference image IMGr. Since the image IMAGaj is an image IMAGa similar to the image IMGa1, when the image IMGa1 is displayed, the information for the operator to judge when attaching the label to the input image IMGi is not displayed even if the image IMAGaj is displayed. But it doesn't change much.

On the other hand, the image display device 200 displays the image IMGa2 satisfying a predetermined positional relationship together with the input image IMGi instead of the image IMGaj. For example, since the images IMGa1 and IMGa2 having vector Va orthogonal to each other on the feature space are selected as the reference image IMGr, the input image IMGi is selected between the reference image IMGr (first image IMG and second image IMG). Features similar to are different.

Thereby, for example, as compared with the case where the images IMGa1, IMGaj, IMGb1 and IMGb2 are selected as the reference image IMGr, it is possible to increase the variation of the feature amount similar to the input image IMGi among the reference image IMGr. As the variation of the feature amount similar to the input image IMGi increases among each reference image IMGr, the operator can easily compare the past labeled image IMG with the input image IMGi. For example, as the variation of the feature amount similar to the input image IMGi increases among each reference image IMGr, the operator can execute the comparison between each reference image IMGr and the input image IMGi from different viewpoints. That is, the image display device 200 allows the operator to assign a label to the input image IMGi by selecting the image IMGa2, as compared with the case where the images IMGa1, IMGaj, IMGb1 and IMGb2 are displayed together with the input image IMGi. Can provide useful information. As a result, the image display device 200 can suppress the occurrence of a contradiction with the past labeling result, as compared with the case where the images IMGa1, IMGaj, IMGb1 and IMGb2 are displayed together with the input image IMGi.

The operation of the image display device 200 is not limited to the example shown in FIG. For example, the image display device 200 may select a plurality of different types of second image IMGs. In this case, the second image specifying unit 230 is the image IMG most similar to the input image IMG among the input image IMGi, the first image IMG, and the image IMG having a predetermined positional relationship with the second image IMG already identified. May be repeated to specify the second image IMG. When the predetermined positional relationship is a positional relationship indicated by a vector V orthogonal to or substantially orthogonal to the vector V indicating the position of the first image IMG, the process of identifying the second image IMG is the number of types of feature quantities. It may be repeated one less number. Further, when the predetermined positional relationship is equal to or greater than the predetermined first image IMG in the feature space, the process of specifying the second image IMG may be repeated by a predetermined number of times. ..

As described above, in the embodiment shown in FIGS. 1 and 2, the image display device 200 calculates a plurality of types of feature amounts from the input image IMGi. Further, the image display device 200 refers to a memory 30 or the like that stores information indicating the type of the image IMG and a plurality of types of feature amounts calculated from the image IMG in association with the image IMG. Then, the image display device 200 identifies the first image IMG that is most similar to the input image IMGi among the image IMGs associated with the first type of the image IMGs. The image display device 200 is an image IMG of the first type among a plurality of image IMGs stored in the memory 30 or the like in the feature space determined by the type of the feature amount, and is an input image IMGi and a first image. A second image IMG having a predetermined positional relationship with the IMG is specified. Then, the image display device 200 displays a screen including information indicating the first type, the first image IMG, the second image IMG, and the input image IMGi.

As a result, the first image IMG and the second image IMG that can be compared with the input image IMGi from different viewpoints are displayed on the display device 110 together with the input image IMGi. As a result, when labeling the image IMG, it is possible to suppress the occurrence of a contradiction with the past labeling result (labeling result of the already labeled image IMG).

FIG. 3 shows another embodiment of an image display program, an image display device, and an image display method. Elements that are the same as or similar to the elements described with reference to FIGS. 1 and 2 are designated by the same or similar reference numerals, and detailed description thereof will be omitted. The image display device 202 shown in FIG. 3 executes, for example, an image display process for suppressing the occurrence of a contradiction with the past labeling result when the operator assigns a label indicating the type of the image IMG to the image IMG. do. The image display device 202 may be realized only by hardware, or may be realized by controlling the hardware by software. In the example shown in FIG. 3, the image display device 202 is realized by the computer 10 executing the image display program DPRG. For example, the computer 10 has a processor 20, a memory 30, a hard disk device 40, an input interface 50, an output interface 60, and a drive device 70. The memory 30 and the hard disk device 40 are examples of storage units.

The image display device 202 is realized, for example, by the processor 20 executing the image display program DPRG stored in the memory 30. For example, a processing device that assigns a label to an image IMG includes a processor 20, a memory 30, a hard disk device 40, an input interface 50, and an output interface 60 that realize the image display device 202.

The image display device 202 has a first image identification unit 222 and a second image identification unit 232 in place of the first image identification unit 220 and the second image identification unit 230 shown in FIG. Further, in the image display device 202, the similarity calculation unit 212 and the registration unit 250 are added to the image display device 200 shown in FIG. Other configurations of the image display device 202 are the same as or similar to those of the image display device 200 shown in FIG.

For example, the image display device 202 has a feature amount calculation unit 210, a similarity calculation unit 212, a first image identification unit 222, a second image identification unit 232, a display unit 240, and a registration unit 250. The feature amount calculation unit 210 is the same as or similar to the feature amount calculation unit 210 shown in FIG. For example, the feature amount calculation unit 210 calculates a plurality of types of feature amounts from the input image IMGi.

The similarity calculation unit 212 refers to the feature amount of the already labeled image IMG and the feature amount database CDB in which the label is registered in association with the image IMG, and refers to the image IMG already labeled as the input image IMGi. Calculate the similarity with. The feature database CDB is stored in, for example, the memory 30. Further, the memory 30 holds an image database IDB in which the image IMG is registered. The feature database CDB, the image database IDB, and the like may be stored in the hard disk device 40.

For example, the similarity calculation unit 212 compares the feature amount calculated from the input image IMGi with the feature amount registered in the feature amount database CDB, and compares the feature amount registered in the feature amount database CDB with the input image IMGi and the image IMG registered in the image database IDB (already labeled). The degree of similarity with the attached image IMG) is calculated. Then, the similarity calculation unit 212 stores the calculated similarity in the memory 30 or the like.

The first image identification unit 222 is the first image identification unit shown in FIG. 1, except that the similarity calculation unit 212 calculates the similarity between the input image IMGi and the already labeled image IMG. Same as or similar to 220. For example, the first image identification unit 222 is the first image IMG having the characteristics most similar to the input image IMGi among the image IMGs registered in the image database IDB based on the similarity calculated by the similarity calculation unit 212. Is specified for each type of image IMG (for each type of label). In this way, the first image identification unit 222 identifies the first image IMG that is most similar to the input image IMGi among the image IMGs associated with the first type of the image IMGs. The similarity calculation unit 212 and the first image identification unit 222 refer to the storage unit that stores the feature amount and the label of the image IMG, and are the most input image IMG among the image IMGs associated with the first type. This is an example of a first image identification unit that identifies a similar first image IMG.

The second image identification unit 232 is the same as or similar to the second image identification unit 230 shown in FIG. For example, the second image identification unit 232 is associated with the first type among the plurality of image IMGs registered in the image database IDB, and is predetermined to be the input image IMGi and the first image IMG in the feature space. The second image IMG in the positional relationship of is specified. The details of the operation of the second image specifying unit 232 will be described with reference to FIG.

The display unit 240 is the same as or similar to the display unit 240 shown in FIG. For example, the display unit 240 transmits the information PINF including the input image IMGi, the reference image IMGr (first image IMG, the second image IMG), and the label of the reference image IMGr to the display device 110 via the output interface 60. Output. As a result, the labels of the input image IMGi, the reference image IMGr, and the reference image IMGr are displayed on the display device 110.

As described above, when the image display device 202 assigns a label to the input image IMG such as the image IMG captured by the image pickup device 90, the image display device 202 has already labeled the reference image IMGr to be displayed on the display device 110 together with the input image IMGi. Select from the created image IMG. Then, the operator refers to the image IMG displayed on the display device 110 and assigns a label to the input image IMGi. Information such as the feature amount of the labeled image IMG is registered in, for example, the feature amount database CDB.

For example, the registration unit 250 receives information indicating the label of the input image IMGi labeled by the operator via the input interface 50. Then, the registration unit 250 registers the feature amount and the label of the input image IMGi in the feature amount database CDB in association with the input image IMGi. The input image IMGi is registered in the image database IDB, for example, when the image display device 202 receives the input image IMGi. Registration of the input image IMGi in the image database IDB may be executed by the registration unit 250, or may be executed by a module other than the registration unit 250.

The configuration of the image display device 202 and the computer 10 is not limited to the example shown in FIG. For example, the image display device 202 may be defined except for the registration unit 250. In this case, for example, the processing device that attaches the label to the image IMG has an image display device 202 excluding the registration unit 250, a registration unit 250, a memory 30, a hard disk device 40, an input interface 50, and an output interface 60.

FIG. 4 shows an example of the feature database CDB shown in FIG. In the feature amount database CDB, data indicating the label number, the image file name, and the feature amount are registered for each image IMG from which the feature amount representing the feature of the object to be inspected is extracted.

The label number is a number corresponding to a label indicating the type of image IMG. For example, the same label number indicates the same type. The image file name is the file name of the image IMG used for extracting the feature amount, and is associated with the file name of the image IMG registered in the image database IDB. The feature amount is a feature amount representing the feature of the object extracted from the image IMG indicated by the image file name.

In the example shown in FIG. 4, shape features (object area, perimeter, etc.), texture features (edge density, luminance histogram, etc.) and the like are registered in the feature database CDB as feature quantities.

The area, perimeter, edge density, etc. are one-dimensional features having one element each, and the luminance histogram is a multidimensional feature having a plurality of classes (elements). The image display device 202 treats the data (for example, frequency) of each dimension of the multidimensional feature amount as one feature amount. In the example shown in FIG. 4, the luminance histogram is a 256-dimensional feature quantity, and the values of each class (1, 2, 3, ..., 256) are treated as one feature quantity. The number of classes in the luminance histogram is not limited to 256. Further, the items registered in the feature amount database CDB are not limited to the example shown in FIG.

FIG. 5 shows an example of the operation of the similarity calculation unit 212 shown in FIG. Note that FIG. 5 shows an example of the operation of the similarity calculation unit 212 when there are two types of feature amounts, feature amount 1 (for example, edge density) and feature amount 2 (for example, area). In this case, the feature space is two-dimensional and has a horizontal axis indicating the feature amount 1 and a vertical axis indicating the feature amount 2. The shaded rectangle shown in FIG. 5 indicates an image IMGa already labeled with a type a (for example, a non-defective product), and the triangle indicates an image already labeled with a type b (for example, a defective product). IMGb is shown, and the star shape indicates the input image IMGi to which the label is attached. In FIG. 5, the operation of the similarity calculation unit 212 will be described by taking as an example the case where the label given to the input image IMGi is either a label indicating the type a or a label indicating the type b. The number of types of image IMG and the type of feature amount are not limited to two.

The similarity calculation unit 212 determines the degree of similarity between the feature amount of the input image IMGi (feature amount calculated by the feature amount calculation unit 210) and the feature amount registered in the feature amount database CDB, and the image registered in the image database IDB. Calculated for each IMG (already labeled image IMG). The similarity is represented by the distance DIS between the input image IMGi on the feature space and the image IMG (IMGa, IMGb) for which the similarity is to be calculated.

The distance DIS is represented by the equation (1) using the coordinates (o1, o2) of the input image IMG on the feature space and the coordinates (c1, c2) of the image IMG for which the similarity is to be calculated. The reference numeral o1 is the value of the feature amount 1 of the input image IMGi, and the reference numeral o2 is the value of the feature amount 2 of the input image IMGi. Similarly, the reference numeral c1 is the value of the feature amount 1 of the image IMG whose similarity is to be calculated, and the reference numeral c2 is the value of the feature amount 2 of the image IMG whose similarity is to be calculated. The sign ^ indicates a power.
DIS = √ ((o1-c1) ^ 2 + (o2-c2) ^ 2) ‥ (1)
The distance DIS indicates that the smaller the value (the shorter the distance between the image IMGs in the feature space), the higher the similarity. Therefore, among the plurality of image IMGs, the image IMG most similar to the input image IMG is the image IMG having the smallest distance DIS.

Further, when the types of the feature amounts of the image IMG are m (m is an integer of 3 or more), the distance DIS is the calculation target of the values o1, o2, ..., Om and the similarity of the plurality of feature amounts of the input image IMGi. Image IMG is represented by the formula (2) using the values c1, c2, ..., Cm of a plurality of feature quantities.
DIS = √ ((o1-c1) ^ 2 + (o2-c2) ^ 2 + ... + (om-cm) ^ 2) ‥ (2)
In this case, the position of the input image IMG in the feature space is represented by the coordinates (o1, o2, ..., Om), and the position of the image IMG for which the similarity is calculated is the coordinates (c1, c2, ..., Om). It is represented by cm). In this way, the similarity calculation unit 212 calculates the distance DIS between the image IMGs on the feature space centered on the type of feature amount (when a multidimensional feature amount is included, each dimension is treated as one type). , Calculated as similarity. In the calculation of the distance DIS using the formula (1) or the formula (2), a weight corresponding to the importance of each feature is added to the feature value (coordinate value of each axis in the feature space), and the weight is given. The value of the added feature quantity may be substituted into the equation (1) or the equation (2).

FIG. 6 shows an example of the operation of the first image identification unit 222 and the second image identification unit 232 shown in FIG. In FIG. 6, the first image identification unit 222 and the second image when there are two types of feature amounts, feature amount 1 and feature amount 2, and the number of image IMG types are type a and type b. An example of the operation of the specific unit 232 is shown. The number of types of image IMG and the types of features are not limited to two. The feature space shown in FIG. 6 is the same as the feature space shown in FIG. Further, the meanings of the shaded quadrangles, triangles, and stars shown in FIG. 6 are the same as those in FIG. Further, the vector V (Va1, Va2, Vaj, Vak, Vb1, Vb2, Vbj) shown in FIG. 6 indicates the position of each image IMG when the position of the input image IMGi in the feature space is used as a starting point. Therefore, the magnitude of the vector V (| Va1 |, etc.) indicates the distance DIS between the input image IMGi and each image IMG in the feature space. Further, the angles θ (θa, θb, θj) shown in FIG. 6 are a vector V indicating the position of the image IMG searched when specifying the second image IMG and a vector V indicating the position of the first image IMG. Indicates the angle between and.

In FIG. 6, the first image has a predetermined positional relationship represented by a vector V orthogonal to a vector V indicating the position of the first image IMG when the position of the input image IMGi in the feature space is used as a starting point. The operations of the specific unit 222 and the second image specific unit 232 will be described.

In the process P1, the first image identification unit 222 is the most input image IMGi (star shape shown in FIG. 6) among the plurality of image IMGa (shaded quadrangle shown in FIG. 6) of type a distributed in the feature space. The close image IMG1 is specified as the first image IMG of type a. In the example shown in FIG. 6, the magnitude | Va1 | of the vector Va1 is smaller than the magnitude | Va2 | of the vector Va2 | and the magnitude | Vaj | of the vector Vaj, and the vector Va corresponding to the plurality of image IMGas of the type a, respectively. Is the smallest of the sizes of. That is, the image IMG1 is an image IMG that is most similar to the input image IMGi among the plurality of image IMGs of type a distributed in the feature space.

Next, in the process P2, the second image identification unit 232 includes the vectors Va2 orthogonal to the vector Va1 indicating the position of the first image IMGa1 among the plurality of image IMGa of the type a, and the images IMGa2 and IMGak represented by Vak. Is extracted. Then, the second image identification unit 232 sets the image IMGa2 closest to the input image IMGi as the second image IMG of the type a among the vectors Va2 orthogonal to the vector Va1, the plurality of images IMGa2 represented by the Vak, and the IMGak. Identify. The magnitude | Va2 | of the vector Va2 is smaller than the magnitude | Vak | of the vector Vak. That is, the image IMGa2 has more similar characteristics to the input image IMGi than the image IMGak.

As described above, the second image identification unit 232 is formed by the vector Va2 having an angle of 90 degrees with the vector Va1 indicating the position of the first image IMGa1 among the plurality of image IMGas of the type a, and the image IMGa2 represented by the Vak. , IMGak is extracted. Then, the second image identification unit 232 identifies the image IMGa2 most similar to the input image IMGi among the extracted plurality of images IMGa2 and IMGak as the second image IMG of the type a.

Here, whether or not the angle formed by the two vectors V is 90 degrees, that is, whether or not the two vectors V are orthogonal to each other is determined by determining whether or not the inner product of the two vectors V becomes 0. Will be done. In the example shown in FIG. 6, since the angle θa formed by the vectors Va1 and Va2 and the angle θa formed by the vectors Va1 and Vak are 90 degrees, the inner products Va1 and Va2 of the vectors Va1 and Va2 and the inner products Va1 and Vak of the vectors Va1 and Vak. Is 0. Therefore, as described above, the second image identification unit 232 extracts the images IMGa2 and IMGak as the image IMG represented by the vector Va orthogonal to the vector Va1 indicating the position of the first image IMGa1. The inner product of the two vectors V is calculated using the equation (5) described later.

As described above, by the process P2, the second image identification unit 232 has a predetermined positional relationship with the input image IMGi and the first image IMGa1 in the feature space among the image IMGa associated with the type a. Is specified as the second image IMG. In the example shown in FIG. 6, the image IMAGaj is similar to the input image IMGi as compared with the image IMGa2, but the positional relationship between the input image IMGi and the first image IMGa1 does not satisfy a predetermined positional relationship, so that the reference image Not selected as IMGr.

Since the reference image IMGr including the first image IMGa1 and the second image IMGa2 is selected in the same number as the type of the feature amount (the number of axes in the feature space), the search for the reference image IMGr for the image IMGa of the type a is performed. ,finish. Next, in the processes P3 and P4, the search for the reference image IMGr for the image IMGb of the type b is executed in the same manner as the search for the reference image IMGr for the image IMGa of the type a.

In the process P3, the first image identification unit 222 is the image IMGb1 closest to the input image IMGi (star shape shown in FIG. 6) among the plurality of image IMGa (triangles shown in FIG. 6) of type b distributed in the feature space. Is specified as the first image IMG of type b.

Next, in the process P4, the second image identification unit 232 selects the image IMGb2 represented by the vector Vb2 orthogonal to the vector Vb1 indicating the position of the first image IMGb1 among the plurality of image IMGb of the type b. Specific to the second image IMG of.

In the example shown in FIG. 6, since the angle θb formed by the vectors Vb1 and Vb2 is 90 degrees, the inner products Vb1 and Vb2 of the vectors Vb1 and Vb2 are 0. Therefore, as described above, the second image identification unit 232 identifies the image IMGb2 represented by the vector Vb2 orthogonal to the vector Vb1 indicating the position of the first image IMGb1 as the second image IMG. Since the angle θj formed by the vector Vbj indicating the position of the image IMGbj and the vector Vb1 is not 90 degrees, the inner product Vb1 and Vbj of the vectors Vb1 and Vbj are not 0. That is, the positional relationship between the input image IMGi and the first image IMGb1 and the image IMGbj does not satisfy a predetermined positional relationship. Therefore, the image IMGbj is similar to the input image IMGi as compared to the image IMGb2, but is not selected as the reference image IMGr.

Since the reference image IMGr including the first image IMGb1 and the second image IMGb2 is selected in the same number as the type of the feature amount (the number of axes in the feature space), the search for the reference image IMGr for the image IMGb of the type b is performed. ,finish. Further, since the reference image IMGr is selected for all types, the search for the reference image IMGr by the first image identification unit 222 and the second image identification unit 232 ends.

As described above, when the label to be given to the input image IMGi is determined to be either the label indicating the type a or the label indicating the type b, the operation shown in FIGS. 5 and 6 is performed together with the input image IMGi. The reference image IMGr to be displayed on the display device 110 is selected. In the operation shown in FIG. 6, for each type of image IMG, the image IMG most similar to the input image IMG is specified as the first image IMG, and the vector V orthogonal to the vector V indicating the position of the first image IMG is used. The image IMG shown is identified as a second image IMG. As a result, as compared with the case where the image IMG similar to the first image IMG is specified as the second image IMG, the input image IMGi is obtained between a plurality of reference image IMGs (first image IMG, second image IMG). It is possible to increase the variation of the feature amount similar to. As a result, in the operation shown in FIG. 6, useful information for the operator to assign a label to the input image IMGi is provided as compared with the case where the image IMG similar to the first image IMG is specified as the second image IMG. Can be provided.

The operations of the first image identification unit 222 and the second image identification unit 232 are not limited to the example shown in FIG. For example, the predetermined positional relationship when specifying the second image IMG is indicated by the vector V within a predetermined first range including 90 degrees at an angle formed by the vector V indicating the position of the first image IMG. It may be a positional relationship. The first range may be, for example, a range of ± 45 degrees (an angle between 90 degrees and 0 degrees) with respect to 90 degrees, or a range that can be regarded as approximately 90 degrees.

For example, when the vector V indicating the position of the first image IMG is Vf and the vector V indicating the position of the image IMG to be determined is Vt, the angle θ formed by the two vectors Vf and Vt is 90 degrees ± de1. Whether or not it is within the range is determined based on the following equations (3) and (4).
cosθ = (Vf · Vt) / (| Vf | × | Vt |) (3)
cos (90 + de1) ≤ cos θ ≤ cos (90-de1) (4)
Vf · Vt of the equation (3) is an inner product of the vectors Vf and Vt, and is the coordinates (o1, o2) of the input image IMG on the feature space, the coordinates (f1, f2) of the first image IMG, and the determination target. It is represented by the equation (5) using the coordinates (t1, t2) of the image IMG. Further, since the respective magnitudes | Vf | and | Vt | of the vectors Vf and Vt are the distance DISs between the input image IMGi and each image IMG in the feature space, the equation (1) described in FIG. 5 is used. Calculated using.
Vf · Vt = (f1-o1) × (t1-o1) ＋ (f2-o2) × (t2-o2) ‥ (5)
When cosθ calculated using the equations (3) and (5) and the like satisfies the equation (4), the second image identification unit 232 has a vector Vf indicating the position of the first image IMG and the image IMG to be determined. It is determined that the angle formed by the vector Vt indicating the position of is within the range of 90 degrees ± de1. That is, the second image identification unit 232 determines that the positional relationship between the input image IMGi and the first image IMG and the image IMG represented by the vector Vt satisfies a predetermined positional relationship.

Further, when the types of the feature amount of the image IMG are m (m is an integer of 3 or more), the inner product Vf · Vt of the vectors Vf and Vt is the number of types of the feature amount (m) in the equation (5). It is represented by the equation (6) extended accordingly. Further, the respective magnitudes | Vf | and | Vt | of the vectors Vf and Vt are calculated using the equation (2) described with reference to FIG.
Vf · Vt = (f1-o1) × (t1-o1) ＋… ＋ (fm-om) × (tm-om) ‥ (6)
The values o1, ..., And om in the equation (6) indicate the respective values of the m types of features of the input image IMG, and the values f1, ..., fm are the features of m types of the first image IMG. Each value of is shown. Further, the values t1, ..., And tm of the equation (6) indicate the respective values of the m types of feature quantities of the image IMG to be determined. In this case, the position of the input image IMGi in the feature space is represented by the coordinates (o1, ..., om), and the position of the first image IMG is represented by the coordinates (f1, ..., fm), and is the judgment target. The position of the image IMG is represented by coordinates (t1, ..., Tm).

FIG. 7 shows an example of the operation of the display unit 240 shown in FIG. Note that FIG. 7 shows an example of the operation of the display unit 240 when there are four types of image IMGs. The number of image IMG types is not limited to four.

The display unit 240 displays the reference image IMGr (first image IMG, second image IMG) specified by the first image identification unit 222 and the second image identification unit 232 on the display device 110. For example, the display unit 240 arranges the input image IMGi in the center of the display screen, and displays the reference image IMGr together for each label around the input image IMGi. As a result, the operator can easily confirm whether or not there is a similar image IMG in the image IMG group to which the label other than the label to be given to the input image IMGi is attached. As a result, when labeling the image IMG, it is possible to suppress the occurrence of a contradiction with the past labeling result (labeling result of the already labeled image IMG).

FIG. 8 shows an example of the operation of the image display device 202 shown in FIG. The operation shown in FIG. 8 is an aspect of the image display method executed by the computer 10. Further, the program for causing the computer 10 to execute the operation shown in FIG. 8 is one aspect of the image display program. The operation shown in FIG. 8 may be realized only by hardware. Detailed description of the same or similar operation as that described with reference to FIG. 2 will be omitted. In the operation shown in FIG. 8, the predetermined positional relationship is the positional relationship indicated by the vector V orthogonal to the vector V indicating the position of the first image IMG when the position of the input image IMGi in the feature space is used as the starting point. be.

In step S10, the image display device 202 determines whether or not the input image IMGi to be labeled exists. For example, the image display device 202 determines whether or not the input image IMGi has been received from the image pickup device 90. When the input image IMGi is present, the operation of the image display device 202 proceeds to step S100. On the other hand, when the input image IMGi does not exist, the operation of the image display device 202 returns to step S10.

In step S100, the feature amount calculation unit 210 calculates the feature amount of the input image IMGi as described with reference to FIG. As a result, the input image IMGi is arranged on the feature space having a plurality of axes corresponding to each of the plurality of types of feature quantities.

Next, in step S200, the similarity calculation unit 212 compares the feature amount of the input image IMGi with the feature amount of another image IMG distributed in the feature space, and compares the feature amount of the input image IMGi with the feature amount of the other image IMG distributed in the feature space, as described in FIG. And the similarity with other image IMGs are calculated for each other image IMG.

Next, in step S310, the first image identification unit 222 sets the variable L indicating the label number to 0 (L = 0).

Next, in step S320, the first image identification unit 222 increments the variable L by 1 (L = L + 1).

Next, in step S330, the first image identification unit 222 sets the image IMG with the label number L as the search target. As a result, in steps S410 and S510, the reference image IMGr (first image IMG, second image IMG) is specified from the image IMG of the type indicated by the label number L among the plurality of types of the image IMG. That is, in step S330, the first image identification unit 222 sets the type indicated by the label number L as the first type among the plurality of types of the image IMG.

Next, in step S410, the first image identification unit 222 sets the variable N to 1 (N = 1). Further, as described with reference to FIG. 6, the first image identification unit 222 selects the image IMG having the maximum similarity among the image IMGs to be searched (the type of image IMG indicated by the label number L) set in step S330. It is specified as the first reference image IMGr (first image IMG). The image IMG having the highest degree of similarity is the image IMG having the smallest distance DIS represented by the formula (1) or the formula (2). After the first reference image IMGr is identified by the process of step S410, the operation of the image display device 202 shifts to step S510.

In step S510, the second image identification unit 232 is orthogonal to all of the N vectors V indicating the positions of the N reference images IMGr in the feature space from the image IMG to be searched set in step S330. The image IMG represented by the vector V is extracted. Then, the second image identification unit 232 identifies the image IMG having the maximum similarity among the extracted image IMGs as the (N + 1) th reference image IMGr. That is, the second image identification unit 232 is the image IMG to be searched, and the similarity among the image IMGs represented by the vectors V orthogonal to all of the N vectors V indicating the positions of the N reference image IMGrs. Specifies the largest image IMG as the (N + 1) th reference image IMGr.

For example, when the variable N is 1, the N reference images IMGr already specified are the first reference image IMGr (first image IMG) specified in step S410. Further, the (N + 1) th reference image IMGr, that is, the second reference image IMGr is the first second image IMG. In this case, as described with reference to FIG. 6, the second image identification unit 232 is derived from the image IMG to be searched, which is represented by a vector V orthogonal to the vector V indicating the position of the first image IMG. The image IMG with the highest similarity is specified as the first second image IMG.

Further, when the variable N is updated to 2 by steps S520 and S530 described later and the process of step S510 is executed again, one of the already identified N (two) reference images IMGr is step S410. The first image IMG identified in. Then, the remaining one of the N (2) reference image IMGs already identified is the first second image IMG identified in the first step S510. The (N + 1) th reference image IMGr, that is, the third reference image IMGr is the second second image IMG. In this case, the second image identification unit 232 is orthogonal to the vector V indicating the position of the first image IMG and indicates the position of the first second image IMG (the second image IMG already specified). The image IMG represented by the vector V orthogonal to the vector V is extracted from the image IMG to be searched. Then, the second image identification unit 232 identifies the image IMG having the maximum similarity among the image IMGs extracted from the image IMGs to be searched as the second second image IMG.

The second image identification unit 232 uses the image IMG to be searched until the image IMG represented by the vector V orthogonal to all of the N vectors V indicating the positions of the N reference images IMGr is extracted. Image IMGs may be extracted in descending order of similarity. In this case, the second image identification unit 232 identifies, for example, the last extracted image IMG as the (N + 1) th reference image IMGr.

After the (N + 1) th reference image IMGr is specified by the process of step S510, the operation of the image display device 202 shifts to step S520. Note that, for example, when an image IMG satisfying a predetermined positional relationship does not exist in the image IMG to be searched, that is, when the second image IMG is not specified, the operation of the image display device 202 is not shown, although it is not shown. , Step S610.

In step S520, the second image identification unit 232 increments the variable N by 1 (N = N + 1).

Next, in step S530, the second image identification unit 232 determines whether or not the variable N is equal to or greater than a predetermined value. The predetermined value is set to, for example, the same value as the number of axes in the feature space, that is, the number of types of feature quantities. The predetermined value may be smaller than the number of axes in the feature space. When the variable N is equal to or greater than a predetermined value, the operation of the image display device 202 proceeds to step S610.

On the other hand, when the variable N is less than a predetermined value, the operation of the image display device 202 returns to step S510. As a result, the process of step S510 is repeated, and a plurality of second image IMGs having the label number L (L is a value set in step S320) are specified.

In step S610, the image display device 202 determines whether or not all the labels have been searched. For example, when the maximum value of the label number registered in the feature amount database CDB and the value of the variable L match, the image display device 202 determines that all the labels have been searched. When all the labels have been searched, the operation of the image display device 202 proceeds to step S710. On the other hand, if there is a label that has not been searched, the operation of the image display device 202 returns to step S320.

In step S710, the display unit 240 displays the input image IMGi and the reference image IMGr of each label on the display device 110 as described with reference to FIG. 7. That is, the display unit 240 displays a screen including the first type, the first image IMG, the second image IMG, and the input image IMGi on the display device 110. Thereby, for example, the operator refers to the image IMG displayed on the display device 110 and assigns a label to the input image IMGi.

Next, in step S800, the image display device 202 determines whether or not the input image IMGi is labeled. For example, when the operator assigns a label to the input image IMGi, the image display device 202 receives information indicating the label of the input image IMGi labeled by the operator via the input interface 50. In this case, the image display device 202 determines that the input image IMGi is labeled. When the input image IMGi is labeled, the operation of the image display device 202 proceeds to step S900. On the other hand, when the input image IMGi is not labeled, the operation of the image display device 202 returns to step S800. For example, the image display device 202 waits for the execution of the process of step S900 until it receives the information indicating the label of the input image IMGi.

In step S900, the registration unit 250 updates the contents of the feature database CDB. For example, the registration unit 250 associates the information indicating the label of the input image IMGi received from the input interface 50 (for example, the label number) with the feature amount of the input image IMGi calculated in step S100 with the input image IMGi to provide a feature amount database. Register with the CDB. As a result, the operation of the image display device 202 when assigning a label to one input image IMGi is completed. Therefore, after updating the contents of the feature amount database CDB, the operation of the image display device 202 returns to step S10.

The operation of the image display device 202 is not limited to the example shown in FIG. For example, in step S510, among the image IMGs to be searched, the angles formed by each of the N vectors V indicating the positions of the N reference images IMGr in the feature space are all within the first range. The image IMG represented by V may be extracted. That is, in the second image identification unit 232, the angles formed by each of the N vectors V indicating the positions of the N reference images IMGr in the feature space of the image IMGs to be searched are all in the first range. The image IMG represented by the vector V inside may be extracted. Then, the second image identification unit 232 may specify the image IMG having the maximum similarity among the extracted image IMGs as the (N + 1) th reference image IMGr. The first range may be, for example, a range of ± 45 degrees with respect to 90 degrees, or a range that can be regarded as approximately 90 degrees, as described with reference to FIG.

Further, in the second image identification unit 232, the image IMG represented by the vector V orthogonal to all of the N vectors V indicating the positions of the N reference images IMGr in the feature space exists in the image IMG to be searched. If not, the image IMG may be extracted using the first range.

Further, the image display program DPRG may be a program for causing the computer 10 to execute a series of processes from step S10 to step S710. In this case, a program other than the image display program DPRG may cause the computer 10 to execute a series of processes from step S800 to step S900.

As described above, even in the embodiments shown in FIGS. 3 to 8, the same effects as those of the embodiments shown in FIGS. 1 and 2 can be obtained. The image display device 202 is a vector V indicating the position of the first image IMG when the position of the input image IMGi in the feature space is the starting point among the image IMGs associated with the first type to be searched. An image IMG represented by a vector V having an angle of 90 degrees is extracted. Alternatively, the image display device 202 is a first range of the image IMGs associated with the type of the search target, in which the angle formed by the vector V indicating the position of the first image IMG in the feature space includes 90 degrees. The image IMG represented by the vector V in is extracted. Then, the image display device 202 identifies the image IMG most similar to the input image IMG among the extracted image IMGs as the second image IMG.

Further, when a plurality of second image IMGs are specified, in addition to the vector V indicating the position of the first image IMG, the angle formed by the vector V indicating the position of the second image IMG already specified is also formed. Be evaluated. For example, in the image display device 202, in addition to the vector V indicating the position of the first image IMG, the angle formed by the vector V indicating the position of the second image IMG already specified is within 90 degrees or the first range. The image IMG represented by the vector V is extracted. Then, the image display device 202 identifies the image IMG most similar to the input image IMG among the extracted image IMGs as the second image IMG and adds it to the reference image IMGr.

Further, the image display device 202 displays the reference image IMGr including the first image IMG and the second image IMG together with the input image IMG for each label. As a result, the first image IMG and the second image IMG that can be compared with the input image IMGi from different viewpoints are displayed together with the input image IMG as the reference image IMGr. Therefore, when labeling the image IMG, it is possible to suppress the occurrence of a contradiction with the past labeling result (labeling result of the already labeled image IMG).

FIG. 9 shows another embodiment of an image display program, an image display device, and an image display method. Elements that are the same as or similar to the elements described with reference to FIGS. 1 to 8 are designated by the same or similar reference numerals, and detailed description thereof will be omitted. The image display device 204 shown in FIG. 9 executes, for example, an image display process for suppressing the occurrence of a contradiction with the past labeling result when the operator assigns a label indicating the type of the image IMG to the image IMG. do. The image display device 204 may be realized only by hardware, or may be realized by controlling the hardware by software. In the example shown in FIG. 9, the image display device 204 is realized by the computer 10 executing the image display program DPRG. For example, the computer 10 has a processor 20, a memory 30, a hard disk device 40, an input interface 50, an output interface 60, and a drive device 70. The memory 30 and the hard disk device 40 are examples of storage units.

The image display device 204 is the same as or similar to the image display device 202 shown in FIG. 3, except that it has a second image identification unit 234 instead of the second image identification unit 232 shown in FIG. For example, the image display device 204 has a feature amount calculation unit 210, a similarity calculation unit 212, a first image identification unit 222, a second image identification unit 234, a display unit 240, and a registration unit 250. The feature amount calculation unit 210, the similarity calculation unit 212, the first image identification unit 222, the display unit 240, and the registration unit 250 are the feature amount calculation unit 210, the similarity calculation unit 212, and the first image identification unit shown in FIG. It is the same as or similar to 222, the display unit 240, and the registration unit 250.

The second image identification unit 234 selects an image IMG whose distance from the first image IMG in the feature space is equal to or greater than a predetermined first distance among the image IMGs associated with the first type to be searched. Extract. Then, the second image identification unit 234 identifies the image IMG most similar to the input image IMG among the extracted image IMGs as the second image IMG. Thereby, as the reference image IMGr displayed together with the input image IMGi, the first image IMG and the second image IMG that can be compared with the input image IMGi from different viewpoints are specified.

The configuration of the image display device 204 and the computer 10 is not limited to the example shown in FIG. For example, the image display device 204 may be defined except for the registration unit 250. In this case, for example, the processing device that attaches the label to the image IMG has an image display device 204 excluding the registration unit 250, a registration unit 250, a memory 30, a hard disk device 40, an input interface 50, and an output interface 60.

FIG. 10 shows an example of the operation of the second image specifying unit 234 shown in FIG. Note that FIG. 10 shows an example of the operation of the second image specifying unit 234 when there are two types of feature amounts, feature amount 1 and feature amount 2, and the number of image IMG types is two, type a and type b. Is shown. The number of types of image IMG and the types of features are not limited to two. The feature space shown in FIG. 10 is the same as the feature space shown in FIG. Further, the meanings of the shaded quadrangles, triangles, and stars shown in FIG. 10 are the same as those in FIG. Further, the distance DIS (DIS2, DISp, DISq) shown in FIG. 10 indicates the distance between the input image IMGi and each image IMG in the feature space. In FIG. 10, the operation of the second image specifying unit 234 will be described with a positional relationship in which the distance from the first image IMG on the feature space is a predetermined first distance D1 or more as a predetermined positional relationship.

The second image identification unit 234 is an image IMGa having a distance from the first image IMGa1 on the feature space of a predetermined first distance D1 or more among the plurality of image IMGas of the type a (the broken line shown in FIG. 10). The image IMGa) outside the circle is extracted. Then, the second image identification unit 234 selects the image IMGa2 closest to the input image IMGi among the plurality of image IMGs whose distance from the first image IMG in the feature space is the first distance D1 or more, of the type a. Specified as the second image IMG. The second image identification unit 234 also executes the search for the second image IMG for the image IMGb of the type b in the same manner as the search for the second image IMG for the image IMGa of the type a.

In the example shown in FIG. 10, the image IMAGaj has a distance from the image IMGa1 less than the first distance D1, and the images IMGa2, IMGap, IMGaq and the like have a distance from the image IMGa1 greater than or equal to the first distance D1. Therefore, the second image identification unit 234 extracts the image IMGa excluding the image IMGa1 and the image IMAGaj from the plurality of image IMGa of the type a. Further, the image IMGa2 is the closest to the input image IMGi among the image IMGa outside the broken line circle shown in FIG. For example, the distance DIS2 between the image IMGa2 and the input image IMGi is shorter than both the distance DISp between the image IMGap and the input image IMGi and the distance DISq between the image IMGaq and the input image IMGi. Therefore, as described above, the second image identification unit 234 is the image closest to the input image IMGi among the plurality of image IMGas having a distance from the first image IMG in the feature space of the first distance D1 or more. IMGa2 is specified as a second image IMG of type a.

Here, the distance between the first image IMG and the image IMG to be extracted is calculated using the formula (1) or the formula (2) described with reference to FIG. In this case, in the description of the formula (1) or the formula (2), the coordinates of the input image IMGi and the coordinates of the image IMG for which the similarity is calculated are the coordinates of the first image IMG and the coordinates of the image IMG to be extracted. Each can be read as a substitute. In the calculation of the distance, a weight corresponding to the importance of each feature is attached to the feature value (coordinate value of each axis in the feature space), and the weighted feature value is given by the equation (1) or It may be substituted into the equation (2).

As described above, the second image identification unit 234 displays the input image IMGi and the image IMGa2 having a predetermined positional relationship with the input image IMG1 and the first image IMGa1 in the feature space among the image IMGa associated with the type a. The image IMG is specified. In the example shown in FIG. 10, the image IMGaj is similar to the input image IMGi as compared with the image IMGa2, but the positional relationship between the input image IMGi and the first image IMGa1 does not satisfy a predetermined positional relationship, so that the reference image Not selected as IMGr.

As a result, as compared with the case where the image IMG similar to the first image IMG is specified as the second image IMG, the input image IMGi is obtained between a plurality of reference image IMGs (first image IMG, second image IMG). It is possible to increase the variation of the feature amount similar to. As a result, in the operation shown in FIG. 10, useful information for the operator to assign a label to the input image IMGi is provided as compared with the case where the image IMG similar to the first image IMG is specified as the second image IMG. Can be provided.

FIG. 11 shows an example of the operation of the image display device 204 shown in FIG. The operation shown in FIG. 11 is an aspect of the image display method executed by the computer 10. Further, the program for causing the computer 10 to execute the operation shown in FIG. 11 is one aspect of the image display program. The operation shown in FIG. 11 may be realized only by hardware. Detailed description of the same or similar operation as that described with reference to FIG. 8 will be omitted. In the operation shown in FIG. 11, the predetermined positional relationship is a positional relationship in which the distance from the first image IMG on the feature space is equal to or greater than the predetermined first distance D1. The operation shown in FIG. 11 is the same as or similar to the operation shown in FIG. 8 except that the process of step S512 is executed instead of the process of step S510 shown in FIG. The predetermined value in step S530 may be a value larger than the number of axes in the feature space. The process of step S512 is executed after the process of step S410 is executed, or when the variable N is determined to be equal to or greater than a predetermined value in step S530.

In step S512, the second image identification unit 234 is an image whose distance from each of the N reference images IMGr in the feature space is equal to or greater than the first distance D1 among the image IMGs to be searched set in step S330. Extract the IMG. Then, the second image identification unit 234 identifies the image IMG having the maximum similarity among the extracted image IMGs as the (N + 1) th reference image IMGr. That is, the second image identification unit 234 is the image IMG to be searched, and has the largest similarity among the image IMGs whose distances from each of the N reference images IMGr in the feature space are equal to or greater than the first distance D1. The image IMG is specified as the (N + 1) th reference image IMGr.

For example, when the variable N is 1, the N reference images IMGr already specified are the first reference image IMGr (first image IMG) specified in step S410. Further, the (N + 1) th reference image IMGr, that is, the second reference image IMGr is the first second image IMG. In this case, as described with reference to FIG. 10, the second image identification unit 234 has a similarity from the image IMG to be searched that has a distance from the first image IMG of the first distance D1 or more. The largest image IMG is specified as the first second image IMG.

Further, when the variable N is updated to 2 in steps S520 and S530 and the process of step S512 is executed again, the already specified N (2) reference images IMGr are the th-orders specified in step S410. One image IMG and the first second image IMG. The (N + 1) th reference image IMGr, that is, the third reference image IMGr is the second second image IMG. In this case, the second image identification unit 234 has a distance from the first image IMG of the first distance D1 or more, and is the same as the first second image IMG (the second image IMG already specified). An image IMG whose distance is equal to or greater than the first distance D1 is extracted from the image IMG to be searched. Then, the second image identification unit 234 identifies the image IMG having the maximum similarity among the image IMGs extracted from the image IMGs to be searched as the second second image IMG.

The second image identification unit 234 extracts the image IMGs whose distances from each of the N reference image IMGrs are equal to or greater than the first distance D1 from the image IMGs to be searched, in descending order of similarity. May be extracted. In this case, the second image identification unit 234 specifies, for example, the last extracted image IMG to the (N + 1) th reference image IMGr.

After the (N + 1) th reference image IMGr is specified by the process of step S512, the operation of the image display device 204 shifts to step S520. Note that, for example, when an image IMG satisfying a predetermined positional relationship does not exist in the image IMG to be searched, that is, when the second image IMG is not specified, the operation of the image display device 204 is not shown, although it is not shown. , Step S610.

As described above, in the operation shown in FIG. 11, the image IMG whose distance from the already specified reference image IMGr is less than the first distance D1 is not selected as the new reference image IMGr. As a result, as compared with the case where the image IMG similar to the first image IMG is specified as the second image IMG, the input image IMGi is obtained between a plurality of reference image IMGs (first image IMG, second image IMG). It is possible to increase the variation of the feature amount similar to.

The operation of the image display device 204 is not limited to the example shown in FIG. For example, in step S530, the image display device 204 searches for an image IMG whose distance from the already specified reference image IMGr is the first distance D1 or more, instead of determining whether or not the variable N is equal to or greater than a predetermined value. It may be determined whether or not it exists in the image IMG. Then, when the image IMG whose distance from the reference image IMGr already specified is equal to or greater than the first distance D1 exists in the image IMG to be searched, the image display device 204 returns the operation to step S512. Further, when the image IMG whose distance from the reference image IMGr already specified is equal to or greater than the first distance D1 does not exist in the image IMG to be searched, the image display device 204 shifts the operation to step S610. Further, for example, when the variable N is 2 or more, the image display device 204 displays an image IMG whose distance DIS from the input image IMGi is a predetermined distance (for example, twice the distance D1 of the first distance) or more. The second image IMG may be excluded from the search target when specifying it.

As described above, even in the embodiments shown in FIGS. 9 to 11, the same effects as those in the embodiments shown in FIGS. 1 to 8 can be obtained. For example, the image display device 204 is associated with the first type, and is the closest to the input image IMG among the image IMGs whose distance to the first image IMG in the feature space is the first distance D1 or more. The image IMG is identified as a second image IMG. As a result, the image display device 204 sets the distance in the feature space between the plurality of reference images IMGr (first image IMG, second image IMG) displayed together with the input image IMGi to the first distance D1 or more. can. That is, a plurality of reference image IMGrs (first image IMG and second image IMG) that can be compared with the input image IMGi from different viewpoints are displayed together with the input image IMGi. Therefore, when labeling the image IMG, it is possible to suppress the occurrence of a contradiction with the past labeling result (labeling result of the already labeled image IMG).

FIG. 12 shows another embodiment of an image display program, an image display device, and an image display method. Elements that are the same as or similar to the elements described with reference to FIGS. 1 to 11 are designated by the same or similar reference numerals, and detailed description thereof will be omitted. The image display device 206 shown in FIG. 12 executes, for example, an image display process for suppressing the occurrence of a contradiction with the past labeling result when the operator assigns a label indicating the type of the image IMG to the image IMG. do. The image display device 206 may be realized only by hardware, or may be realized by controlling the hardware by software. In the example shown in FIG. 12, the image display device 206 is realized by the computer 10 executing the image display program DPRG. For example, the computer 10 has a processor 20, a memory 30, a hard disk device 40, an input interface 50, an output interface 60, and a drive device 70. The memory 30 and the hard disk device 40 are examples of storage units.

The image display device 206 is the same as or similar to the image display device 204 shown in FIG. 9, except that it has a second image identification unit 236 instead of the second image identification unit 234 shown in FIG. For example, the image display device 206 has a feature amount calculation unit 210, a similarity calculation unit 212, a first image identification unit 222, a second image identification unit 236, a display unit 240, and a registration unit 250. The feature amount calculation unit 210, the similarity calculation unit 212, the first image identification unit 222, the display unit 240, and the registration unit 250 are the feature amount calculation unit 210, the similarity calculation unit 212, and the first image identification unit shown in FIG. It is the same as or similar to 222, the display unit 240, and the registration unit 250.

The second image identification unit 236 excludes an image IMG whose distance DIS from the input image IMGi is equal to or greater than a predetermined second distance from the search target when specifying the second image IMG. It is the same as or similar to the second image specifying unit 234 shown in. For example, the second image specifying unit 236 is the first type of image IMG, the distance to the first image IMG and the like already specified is the first distance D1 or more, and the distance DIS to the input image IMG is the second. Among the image IMGs less than the distance of, the image IMG most similar to the input image IMGi is selected. Then, the second image identification unit 236 identifies the selected image IMG as the second image IMG. The second distance is, for example, a value larger than the first distance D1 and is preset.

The configuration of the image display device 206 and the computer 10 is not limited to the example shown in FIG. For example, the image display device 206 may be defined except for the registration unit 250. In this case, for example, the processing device that attaches the label to the image IMG has an image display device 206 excluding the registration unit 250, a registration unit 250, a memory 30, a hard disk device 40, an input interface 50, and an output interface 60.

FIG. 13 shows an example of the operation of the second image specifying unit 236 shown in FIG. Note that FIG. 13 shows an example of the operation of the second image specifying unit 236 when there are two types of feature amounts, feature amount 1 and feature amount 2, and the number of image IMG types is two, type a and type b. Is shown. The number of types of image IMG and the types of features are not limited to two. The feature space shown in FIG. 13 is the same as the feature space shown in FIG. Further, the meanings of the shaded quadrangle, triangle, star shape, and distance DIS (DIS2, DISp, DISq) shown in FIG. 13 are the same as those in FIG. In FIG. 13, the operation of the second image specifying unit 236 will be described with a positional relationship in which the distance from the first image IMG on the feature space is a predetermined first distance D1 or more as a predetermined positional relationship.

The second image identification unit 236 uses the image IMG (the image IMG outside the dotted circle shown in FIG. 13) having a distance DIS of the input image IMGi of the predetermined second distance D2 or more as the second image IMG. Exclude from the search target when specifying. Other operations of the second image specifying unit 236 are the same as or similar to the operations of the second image specifying unit 234 described with reference to FIGS. 10, 11 and the like.

In the example shown in FIG. 13, the image IMGar is excluded from the search target when specifying the second image IMG of the type a. Thereby, it is possible to prevent the image IMG that is not similar to the input image IMGi from being identified as the second image IMG. That is, it is possible to prevent an image IMG that is not similar to the input image IMGi from being displayed together with the input image IMGi.

Further, the operation of the image display device 206 is that, in step S512 shown in FIG. 11, the image IMG whose distance DIS from the input image IMGi is equal to or greater than the predetermined second distance D2 is excluded from the image IMG to be searched. Except, the operation is the same as or similar to that shown in FIG.

The operations of the second image specifying unit 236 and the image display device 206 are not limited to the example shown in FIG. For example, when the variable N is 1, the image display device 206 may include an image IMG whose distance DIS from the input image IMGi is a predetermined second distance D2 or more as a search target. Further, for example, in step S530 shown in FIG. 11, does the image display device 206 include an image IMG whose distance from the reference image IMGr already specified is equal to or greater than the first distance D1 in the image IMG to be searched? It may be determined whether or not.

As described above, even in the embodiments shown in FIGS. 12 to 13, the same effects as those in the embodiments shown in FIGS. 1 to 11 can be obtained. For example, the image display device 206 is associated with the first type, and searches for an image IMG whose distance from the input image IMGi on the feature space is less than the second distance D2. Then, the image display device 206 is the image IMG to be searched, and among the image IMGs whose distance from the first image IMG in the feature space is the first distance D1 or more, the image IMG closest to the input image IMG is selected. It is specified in the image IMG of 2. As a result, the image display device 206 sets the distance in the feature space between the plurality of reference images IMGr (first image IMG, second image IMG) displayed together with the input image IMGi to the first distance D1 or more. can. That is, a plurality of reference image IMGrs (first image IMG and second image IMG) that can be compared with the input image IMGi from different viewpoints are displayed together with the input image IMGi. Therefore, when labeling the image IMG, it is possible to suppress the occurrence of a contradiction with the past labeling result (labeling result of the already labeled image IMG).

Further, since the image IMG having a distance DIS of the second distance D2 or more from the input image IMGi is not specified as the second image IMG, the image IMG that is not similar to the input image IMGi is displayed together with the input image IMGi. Can be deterred.

FIG. 14 shows another embodiment of an image display program, an image display device, and an image display method. Elements that are the same as or similar to the elements described with reference to FIGS. 1 to 13 are designated by the same or similar reference numerals, and detailed description thereof will be omitted. The image display device 208 shown in FIG. 14 executes, for example, an image display process for suppressing the occurrence of a contradiction with the past labeling result when the operator assigns a label indicating the type of the image IMG to the image IMG. do. The image display device 208 may be realized only by hardware, or may be realized by controlling the hardware by software. In the example shown in FIG. 14, the image display device 208 is realized by the computer 10 executing the image display program DPRG. For example, the computer 10 has a processor 20, a memory 30, a hard disk device 40, an input interface 50, an output interface 60, and a drive device 70. The memory 30 and the hard disk device 40 are examples of storage units.

The image display device 208 has a display unit 242 instead of the display unit 240 shown in FIG. 3, except that a third image identification unit 238 is added to the image display device 202 shown in FIG. It is the same as or similar to the image display device 202 shown in 3. For example, the image display device 208 has a feature amount calculation unit 210, a similarity calculation unit 212, a first image identification unit 222, a second image identification unit 232, a third image identification unit 238, a display unit 242, and a registration unit 250. .. The feature amount calculation unit 210, the similarity calculation unit 212, the first image identification unit 222, the second image identification unit 232, and the registration unit 250 are the feature amount calculation unit 210, the similarity calculation unit 212, and the first It is the same as or similar to the image identification unit 222, the second image identification unit 232, and the registration unit 250.

The third image identification unit 238 is located on a straight line passing through the target reference image IMGr and the input image IMGi in the feature space, and identifies an image of a type different from the target reference image IMGr as the third image IMG. .. The reference image IMGr of interest is, for example, one image IMG of the first image IMG and the second image IMG. The details of the operation of the third image specifying unit 238 will be described with reference to FIG.

The display unit 242 displays the third image IMG as the reference image IMGr on the display device 110 together with the input image IMG in addition to the first image IMG and the second image IMG. It is the same as or similar to the display unit 240 shown in 1. For example, the display unit 242 includes information indicating the first type, information indicating the type of the third image IMG, the first image IMG, the second image IMG, the third image IMG, and the input image IMGi. Display the screen.

The configuration of the image display device 208 and the computer 10 is not limited to the example shown in FIG. For example, the image display device 208 may have the second image identification unit 234 shown in FIG. 9 instead of the second image identification unit 232, or may have the second image identification unit 236 shown in FIG. You may. Further, for example, the image display device 208 may be defined except for the registration unit 250. In this case, for example, the processing device that attaches the label to the image IMG has an image display device 206 excluding the registration unit 250, a registration unit 250, a memory 30, a hard disk device 40, an input interface 50, and an output interface 60.

FIG. 15 shows an example of the operation of the third image identification unit 238 shown in FIG. Note that FIG. 15 shows an example of the operation of the third image specifying unit 238 when there are two types of feature amounts, feature amount 1 and feature amount 2, and the number of image IMG types is two, type a and type b. Is shown. The number of types of image IMG and the types of features are not limited to two. The feature space shown in FIG. 15 is the same as the feature space shown in FIG. The shaded quadrangles, triangles, and stars shown in FIG. 15 have the same meanings as those in FIG. In FIG. 15, it is assumed that the first image IMGa1 and the second image IMGa2 of the type a and the first image IMGb1 and the second image IMGb2 of the type b are specified by the operation shown in FIG. The operation of the image specifying unit 238 will be described.

The third image identification unit 238 displays the image IMGb31 of the type b (an image IMG of a type different from the type a) located on a straight line connecting the first image IMGa1 of the type a and the input image IMGi as a third image. Specific to IMG. As a result, the third image IMGb31 is added as a reference image IMGr having the same viewpoint as the first image IMGa1 of the type a.

Further, the third image identification unit 238 displays the image IMGb32 of the type b (an image IMG of a type different from the type a) located on a straight line connecting the second image IMGa2 of the type a and the input image IMGi. Specific to the image IMG. As a result, the third image IMGb32 is added as a reference image IMGr having the same viewpoint as the second image IMGa2 of the type a.

Similarly, the third image identification unit 238 identifies the third image IMGa31 as the reference image IMGr having the same viewpoint as the first image IMGb1 of the type b, and the third image IMGa32 is the second image IMGa32 of the type b. It is specified as a reference image IMGr having the same viewpoint as the image IMGb2. As described above, the third image identification unit 238 has a different type of reference image IMGr having the same viewpoint as each reference image IMGr for each reference image IMGr specified by the first image identification unit 222 and the second image identification unit 232. To add.

The first image IMG, the second image IMG, and the third image IMG are displayed together with the input image IMG as the reference image IMGr. Thereby, for example, it is possible to provide information capable of confirming whether the input image IMGi is closer to the image IMGa of the type a or the image IMGb of the type b from the viewpoint on the axis of the dotted line shown in FIG.

The operation of the third image specifying unit 238 is not limited to the example shown in FIG. For example, the straight line (dotted line in FIG. 15) passing through each of the first image IMG and the second image IMG and the input image IMGi may have a predetermined width. For example, when the straight line passing through each of the first image IMG and the second image IMG and the input image IMGi has a width wider than the dotted straight line shown in FIG. 15, the image IMG near the dotted straight line shown in FIG. 15 is used. It can be extracted as a third image IMG. As a result, even if an image IMG of a different type from the first image IMG or the second image IMG does not exist on the straight line of the dotted line shown in FIG. It can be specified as an image IMG.

Further, the third image identification unit 238 is indicated by a vector V having an angle formed by a vector V indicating the position of the target reference image IMGr in the feature space of a predetermined angle or more and 180 degrees or less, and the target reference image IMGr. An image IMG of a different type from the above may be specified as a third image IMG. The reference image IMGr of interest is, for example, one image IMG of the first image IMG and the second image IMG.

FIG. 16 shows an example of the operation of the display unit 242 shown in FIG. Note that FIG. 16 shows an example of the operation of the display unit 242 when there are two types of image IMGs. The number of image IMG types is not limited to two. The images IMGa1, IMGb1, IMGa2, IMGb2, IMGa31, IMGa32, IMGb31, and IMGb32 are, for example, the reference image IMGr (first image IMG, second image IMG, third image IMG) described with reference to FIG.

The display unit 242 displays the reference image IMGr specified by the first image identification unit 222, the second image identification unit 232, and the third image identification unit 238 on the display device 110. For example, the display unit 242 arranges the input image IMGi in the center of the display screen, and displays the reference image IMGr on the left and right sides of the input image IMGi for each label.

In the example shown in FIG. 16, among the images IMGa of the label a, the images IMGa1 and IMGa2 similar to the input image IMGi are displayed first and second from the top in the display area of the image IMGa of the label a. .. Then, in the display area of the image IMGb of the label b, the image IMGb31 having the same viewpoint as the image IMGa1 is displayed first from the top, and the image IMGb32 having the same viewpoint as the image IMGa2 is displayed second from the top. Will be done. Therefore, among the image IMGb of the label b, the images IMGb1 and IMGb2 similar to the input image IMGi are displayed as the third and fourth images from the top in the display area of the image IMGb of the label b, respectively. Therefore, in the display area of the image IMGa of the label a, the image IMGa31 having the same viewpoint as the image IMGb1 is displayed third from the top, and the image IMGa32 having the same viewpoint as the image IMGb2 is displayed fourth from the top. Will be done.

In this way, the display unit 242 displays the image IMG (the image IMG located on the same line passing through the input image IMGi in the feature space) having the same viewpoints of the feature amount on the same line. As a result, for example, when an image IMG similar to the input image IMG exists among the labels other than the label to be given to the input image IMGi, the operator considers the correct answer when viewed from the same viewpoint in the label. The image IMG can be easily confirmed. That is, it becomes possible to easily confirm which label the input image IMGi is closer to. As a result, when labeling the image IMG, it is possible to suppress the occurrence of a contradiction with the past labeling result (labeling result of the already labeled image IMG).

FIG. 17 shows an example of the operation of the image display device 208 shown in FIG. The operation shown in FIG. 17 is one aspect of the image display method executed by the computer 10. Further, the program for causing the computer 10 to execute the operation shown in FIG. 17 is one aspect of the image display program. The operation shown in FIG. 17 may be realized only by hardware. Detailed description of the same or similar operation as that described with reference to FIG. 8 will be omitted. The operation shown in FIG. 17 is shown in FIG. 8 except that the process of step S620 is added to the operation shown in FIG. 8 and the process of step S712 is executed instead of the process of step S710 shown in FIG. Same or similar to the shown operation. The process of step S620 is executed when it is determined in step S610 that all the labels have been searched.

In step S620, the third image identification unit 238 adds the third image IMG as the reference image IMGr, as described with reference to FIG. For example, the third image identification unit 238 has a different label image IMG having the same viewpoint as the reference image IMGr for each reference image IMGr (first image IMG, second image IMG) specified in steps S410 and S510. Is specified as the third image IMG. As a result, the third image IMG is displayed on the display device 110 together with the input image IMGi and the like in step S712 described later. After the process of step S620 is executed, the operation of the image display device 208 shifts to step S712.

In step S712, the display unit 242 displays the input image IMGi and the reference image IMGr of each label on the display device 110 as described with reference to FIG. That is, the display unit 242 displays a screen including the first type, information indicating the type of the third image IMG, the first image IMG, the second image IMG, the third image IMG, and the input image IMGi. Display on 110. As a result, since the image IMG having the same viewpoints of the reference image IMGr is displayed on all the labels, it is possible to easily grasp the sense of distance between the input image IMGi and the reference image IMGr of each label from a certain viewpoint. After the process of step S712 is executed, the operation of the image display device 208 shifts to step S800.

The operation of the image display device 208 is not limited to the example shown in FIG. For example, the image display device 208 may execute the process of step S512 shown in FIG. 11 instead of the process of step S510.

As described above, even in the embodiments shown in FIGS. 14 to 17, the same effects as those in the embodiments shown in FIGS. 1 to 13 can be obtained. For example, the image display device 208 is an image IMG to be searched, and is an input image among the image IMGs whose angle formed by the vector V indicating the position of the first image IMG in the feature space is 90 degrees. The image IMG most similar to the IMGi is identified as the second image IMG.

Further, the image display device 208 performs a process of identifying an image IMG having a different label having the same viewpoint as one of the first image IMG and the second image IMG into the third image IMG. This is performed for each of the image IMG and the second image IMG. Then, the image display device 208 displays the input image IMGi and the reference image IMGr (first image IMG, second image IMG, third image IMG) of each label on the display device 110.

As a result, since the image IMG having the same viewpoints of the reference image IMGr is displayed on all the labels, it is possible to easily grasp the sense of distance between the input image IMGi and the reference image IMGr of each label from a certain viewpoint. As a result, when labeling the image IMG, it is possible to suppress the occurrence of a contradiction with the past labeling result (labeling result of the already labeled image IMG).

The inventions described in the above embodiments are organized and disclosed as an appendix as follows.
(Appendix 1)
Calculate multiple types of features from the input image
Of the images associated with the first type of the above types, with reference to the storage unit that stores the information indicating the type of the image associated with the image and the plurality of types of feature quantities calculated from the image. Identify the first image that most closely resembles the input image and
In the feature space determined by the type of feature amount, among the plurality of images stored in the storage unit, the first type is associated with the input image and the first image and a predetermined image. Identify the second image in the positional relationship,
An image display program for causing a computer to execute a process of displaying a screen including information indicating the first type, the first image, the second image, and the input image.
(Appendix 2)
In the image display program described in Appendix 1,
Among the images associated with the first type, a vector having an angle of 90 degrees with a vector indicating the position of the first image when the position of the input image in the feature space is used as a starting point. An image display program for causing a computer to execute a process of identifying the shown image as the second image.
(Appendix 3)
In the image display program described in Appendix 2,
When a plurality of images represented by a vector having an angle of 90 degrees with the vector indicating the position of the first image are extracted, the image most similar to the input image is selected from the extracted plurality of images. An image display program that allows a computer to perform processing that identifies an image.
(Appendix 4)
In the image display program described in Appendix 3,
Further, when a plurality of the second images are specified, the position of the first image is associated with the first type and the position of the first image when the position of the input image in the feature space is used as a starting point. Of the images represented by the vector in which the angle formed by each of the indicated vector and the already identified vector indicating the position of the second image is 90 degrees, the image most similar to the input image is the second image. An image display program that allows a computer to perform the processing specified in.
(Appendix 5)
In the image display program described in Appendix 1,
A predetermined first type that is associated with the first type and has an angle of 90 degrees with a vector indicating the position of the first image when the position of the input image in the feature space is used as a starting point. Among the images represented by the vectors within the range of 1, the image most similar to the input image is specified as the second image.
Further, when a plurality of the second images are specified, the position of the first image is associated with the first type and the position of the first image when the position of the input image in the feature space is used as a starting point. Among the images represented by the vector in which the angles formed by each of the indicated vector and the already identified vector indicating the position of the second image are all within the first range, the image most similar to the input image is described. An image display program for causing a computer to execute a process that identifies a second image.
(Appendix 6)
In the image display program described in Appendix 1,
Among the images associated with the first type and having a distance from the first image on the feature space equal to or greater than a predetermined first distance, the image most similar to the input image is the second image. An image display program that allows a computer to perform processing that identifies an image in.
(Appendix 7)
In the image display program described in Appendix 6,
Further, when a plurality of the second images are specified, the distance between the first image and the already specified second image in the feature space is associated with the first type. Is an image display program for causing a computer to execute a process of identifying an image most similar to the input image as the second image among the images of the first distance or more.
(Appendix 8)
In the image display program described in Appendix 6 or Appendix 7,
An image display program for causing a computer to execute a process of excluding an image whose distance from the input image is equal to or greater than a predetermined second distance from the search target when specifying the second image.
(Appendix 9)
In the image display program according to any one of Appendix 1 to Appendix 8,
A third image is an image of a type different from the one image, which is located on a straight line passing through the first image and one of the second images and the input image in the feature space. Specific to
A process of displaying a screen including the information indicating the first type, the information indicating the type of the third image, the first image, the second image, the third image, and the input image. An image display program to be executed by a computer.
(Appendix 10)
In the image display program described in Appendix 9,
An image display program characterized in that the straight line has a predetermined width.
(Appendix 11)
In the image display program according to any one of Appendix 1 to Appendix 8,
When the angle formed by the vector indicating the position of one of the first image and the second image when the position of the input image is set as the starting point in the feature space is 180 degrees at a predetermined angle or more. An image of a type different from the one image shown by the following vector is specified as a third image.
A process of displaying a screen including the information indicating the first type, the information indicating the type of the third image, the first image, the second image, the third image, and the input image. An image display program to be executed by a computer.
(Appendix 12)
In the image display program according to any one of Appendix 1 to Appendix 11,
An image for changing the first type to another type among the types and causing a computer to execute a process of specifying the first image and specifying the second image for each type. Display program.
(Appendix 13)
A feature amount calculation unit that calculates multiple types of feature amounts from the input image,
Of the images associated with the first type of the above types, with reference to the storage unit that stores the information indicating the type of the image associated with the image and the plurality of types of feature quantities calculated from the image. A first image identification unit that identifies a first image that most closely resembles the input image,
In the feature space determined by the type of feature amount, among the plurality of images stored in the storage unit, the first type is associated with the input image and the first image and a predetermined image. A second image identification unit that identifies a second image that is in a positional relationship,
An image display device including an information indicating the first type, a display unit for displaying a screen including the first image, the second image, and the input image.
(Appendix 14)
An image display method performed by a computer
The computer
Calculate multiple types of features from the input image
Of the images associated with the first type of the above types, with reference to the storage unit that stores the information indicating the type of the image associated with the image and the plurality of types of feature quantities calculated from the image. Identify the first image that most closely resembles the input image and
In the feature space determined by the type of feature amount, among the plurality of images stored in the storage unit, the first type is associated with the input image and the first image and a predetermined image. Identify the second image in the positional relationship,
An image display method comprising displaying a screen including information indicating the first type, the first image, the second image, and the input image.
(Appendix 15)
Calculate multiple types of features from the input image
Of the images associated with the first type of the above types, with reference to the storage unit that stores the information indicating the type of the image associated with the image and the plurality of types of feature quantities calculated from the image. Identify the first image that most closely resembles the input image and
In the feature space determined by the type of feature amount, among the plurality of images stored in the storage unit, the first type is associated with the input image and the first image and a predetermined image. Identify the second image in the positional relationship,
It is characterized in that an image display program for causing a computer to execute a process of displaying a screen including the information indicating the first type, the first image, the second image, and the input image is recorded. A computer-readable recording medium.

The above detailed description will clarify the features and advantages of the embodiments. It is intended that the claims extend to the features and advantages of the embodiments as described above, without departing from their spirit and scope of rights. Also, anyone with ordinary knowledge in the art should be able to easily come up with any improvements or changes. Therefore, there is no intention to limit the scope of the embodiments having invention to those described above, and it is possible to rely on suitable improvements and equivalents included in the scope disclosed in the embodiments.

10 Computer; 20 Processor; 30 Memory; 40 Hard disk device; 50 Input interface; 60 Output interface; 70 Drive device; 80 Bus; 90 Imaging device; 100 Input device; 110 Display device 120 ... Removable media; 200, 202, 204, 206, 208 ... Image display device; 210 ... Feature amount calculation unit; 212 ... Similarity calculation unit; 220, 222 ... First image identification unit; 230, 232, 234, 236 ... 2nd image identification unit; 238 ... 3rd image identification unit; 240, 242 ... Display unit; 250 ... Registration unit

Claims (11)

Calculate multiple types of features from the input image
Of the images associated with the first type of the above types, with reference to the storage unit that stores the information indicating the type of the image associated with the image and the plurality of types of feature quantities calculated from the image. Identify the first image that most closely resembles the input image and
Among the plurality of images stored in the storage unit on the coordinate space centered on each type of feature amount, the first type is associated with the input image and the first image. Identify the second image that has a predetermined positional relationship,
An image display program for causing a computer to execute a process of displaying a screen including information indicating the first type, the first image, the second image, and the input image. In the image display program according to claim 1,
Among the images associated with the first type, a vector having an angle of 90 degrees with a vector indicating the position of the first image when the position of the input image in the coordinate space is used as a starting point. An image display program for causing a computer to execute a process of identifying the shown image as the second image. In the image display program according to claim 2,
When a plurality of images represented by a vector having an angle of 90 degrees with the vector indicating the position of the first image are extracted, the image most similar to the input image is selected from the extracted plurality of images. An image display program that allows a computer to perform processing that identifies an image. In the image display program according to claim 3,
Further, when a plurality of the second images are specified, the position of the first image is associated with the first type and the position of the first image when the position of the input image in the coordinate space is used as a starting point. Among the images represented by the vector in which the angle formed by each of the indicated vector and the already identified vector indicating the position of the second image is 90 degrees, the image most similar to the input image is the second image. An image display program that allows a computer to perform the process specified in. In the image display program according to claim 1,
A predetermined first type that is associated with the first type and has an angle of 90 degrees with a vector indicating the position of the first image when the position of the input image in the coordinate space is used as a starting point. Among the images represented by the vectors within the range of 1, the image most similar to the input image is specified as the second image.
Further, when a plurality of the second images are specified, the position of the first image is associated with the first type and the position of the first image when the position of the input image in the coordinate space is used as a starting point. Among the images represented by the vector in which the angles formed by each of the indicated vector and the vector indicating the position of the second image already specified are all within the first range, the image most similar to the input image is described. An image display program for causing a computer to execute a process that identifies a second image. In the image display program according to claim 1,
Among the images associated with the first type and having a distance to the first image in the coordinate space equal to or greater than a predetermined first distance, the image most similar to the input image is the second image. An image display program that allows a computer to perform processing that identifies an image in. In the image display program according to claim 6,
Further, when a plurality of the second images are specified, the distance between the first image and the already specified second image in the coordinate space is associated with the first type. Is an image display program for causing a computer to execute a process of identifying an image most similar to the input image as the second image among the images of the first distance or more. In the image display program according to claim 6 or 7.
An image display program for causing a computer to execute a process of excluding an image whose distance from the input image is equal to or greater than a predetermined second distance from the search target when specifying the second image. In the image display program according to any one of claims 1 to 8.
A third image is an image of a type different from the one image, which is located on a straight line passing through the first image and one of the second images and the input image in the coordinate space. Specific to
A process of displaying a screen including the information indicating the first type, the information indicating the type of the third image, the first image, the second image, the third image, and the input image. An image display program to be executed by a computer. A feature amount calculation unit that calculates multiple types of feature amounts from the input image,
Of the images associated with the first type of the above types, with reference to the storage unit that stores the information indicating the type of the image associated with the image and the plurality of types of feature quantities calculated from the image. A first image identification unit that identifies a first image that most closely resembles the input image,
Among the plurality of images stored in the storage unit on the coordinate space centered on each type of feature amount, the first type is associated with the input image and the first image. A second image identification unit that specifies a second image having a predetermined positional relationship,
An image display device including an information indicating the first type, a display unit for displaying a screen including the first image, the second image, and the input image. An image display method performed by a computer
The computer
Calculate multiple types of features from the input image
Of the images associated with the first type of the above types, with reference to the storage unit that stores the information indicating the type of the image associated with the image and the plurality of types of feature quantities calculated from the image. Identify the first image that most closely resembles the input image and
Among the plurality of images stored in the storage unit on the coordinate space centered on each type of feature amount, the first type is associated with the input image and the first image. Identify the second image that has a predetermined positional relationship,
An image display method comprising displaying a screen including information indicating the first type, the first image, the second image, and the input image.

Images