JP7618862B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to technology used for inspection work of objects using images.

Inspection work for inspecting structures and other objects involves going to the site and taking photographs of the structure. After that, returning to the office, the photographed images are carefully observed to determine any abnormalities such as cracks, lifting, or peeling on the concrete surface, and an inspection report is created. Determining abnormalities here refers to the act of identifying the type and extent of the abnormality seen in the images. For example, determining that a certain abnormality is a "crack" and that the extent of the abnormality is "the crack width is 0.2 (mm)".

When assessing abnormalities, even if the abnormalities are of the same type and degree, the way they appear can vary greatly depending on the material of the subject and the shooting environment, making it difficult for inspectors to have a consistent standard for their assessment. Also, when the structure being inspected is large, the amount of images that inspectors must review tends to be enormous. For these reasons, it is difficult to continue to consistently assess abnormalities in structure inspection work, and the assessment results are prone to variation.

In response to the issue of maintaining consistency in judgment criteria when determining abnormalities and making diagnoses by observing such images, Patent Document 1 discloses a method for comparing an inspection image with a reference image that is used as a comparison target in the inspection of electronic components. In addition, Patent Document 1 also discloses obtaining parameters for adjusting the appearance from the reference image, and taking an inspection image so as to match the appearance with the reference image when taking the image, making it easier to compare the two. Patent Document 2 discloses a technology that holds parameters for adjusting the appearance of lesions from CT (computed tomography) images as presets, and adjusts the appearance of a specified region of interest by applying the presets, making it easier to make consistent judgments.

JP 2016-65875 A JP 2002-165786 A

However, the technology disclosed in Patent Document 1 requires the preparation of reference images for each type and level of abnormality, which is time-consuming. Similarly, the technology disclosed in Patent Document 2 requires the preparation of parameter presets in advance, which is also time-consuming.

Therefore, the present invention aims to make it possible to reduce the effort required prior to inspection work.

The present invention comprises an acquisition means for acquiring a first captured image of a structure, a display control means for classifying a plurality of second captured images of the structure according to the width of the deformation of the structure and displaying them on a display unit, a selection means for selecting one of the second captured images based on a user operation, and a determination means for determining the width of the deformation of the structure included in the first captured image based on the second captured image selected by the selection means, wherein the display control means displays the first captured image and the second captured image selected by the selection means on the same screen , and when a zoom magnification is changed based on a user operation, the display control means adjusts the ratio of pixel units to distance units for the first captured image and the second captured image selected by the selection means, and displays the first captured image with its position adjusted so that the partial image that was displayed immediately before the zoom magnification was changed is at the center, and displays the second captured image with its position adjusted so that the deformation is included in the displayed partial area .

The present invention makes it possible to reduce the amount of work required prior to inspection work.

1 is a diagram illustrating an overall configuration of an information processing apparatus according to an embodiment. FIG. 2 is a functional block diagram of a structure inspection process according to the first embodiment. FIG. 4 illustrates a table for storing reference information according to the first embodiment; FIG. 2 is a diagram illustrating a screen configuration according to the first embodiment. 11A to 11C are diagrams illustrating alternative examples of the screen configuration of the first embodiment. 13 is a flowchart of an image adjustment process according to reference information selection. 13 is a flowchart illustrating a reference information generating process. FIG. 11 is a functional block diagram of a structure inspection process according to a second embodiment. 11 is a flowchart of an image adjustment process in response to a user operation on a target image. FIG. 13 is a functional block diagram of a structure inspection process according to a third embodiment. FIG. 13 illustrates a table for storing reference information according to the third embodiment. FIG. 13 is a diagram illustrating a screen configuration according to a third embodiment. 13 is a flowchart of an image adjustment process according to a third embodiment. 11A to 11C are diagrams illustrating an example of a transition of a display state. 11A to 11C are diagrams illustrating an example of a transition of a display state.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to those configurations. In this embodiment, an example of a structure inspection task is described, in which an image of a structure such as a building as an inspection target is used to inspect the structure for abnormalities.
First Embodiment
FIG. 1 is a schematic hardware configuration diagram of an information processing device according to a first embodiment. The CPU 101 is a central processing unit that controls the computer system. The CPU 101 performs calculations and processing of information, controls each hardware, and the like based on a control program and a processing program according to this embodiment, thereby realizing various processes including a structure inspection process described later and the configuration of each functional block. The RAM 102 is a random access memory, and functions as a main memory of the CPU 101, as a work memory required for loading an execution program and executing the program. The ROM 103 is a read-only memory that records a control program that specifies the operation processing procedure of the CPU 101 and a processing program according to this embodiment. The ROM 103 includes a program ROM that records a basic software (OS), which is a system program that controls the equipment of the computer system, and a data ROM that records information necessary to operate the system. In some cases, the HDD 107 described later is used instead of the ROM 103. The NETIF 104 is a network interface that controls input and output of data such as images transmitted and received via a network. The program according to this embodiment may be downloaded via the NETIF 104 and recorded in the HDD 107 or the like. The display device 105 is, for example, a CRT display or a liquid crystal display. The input device 106 is an operation input unit for receiving operation instructions from a user, and is, for example, a touch panel, a keyboard, a mouse, or the like. The HDD 107 is a hard disk drive and a storage device. The HDD 107 is used for storing application programs and data such as images. The bus 108 is an input/output bus (address bus, data bus, and control bus) for connecting the above-mentioned units.

Figure 2 is a diagram showing functional blocks that realize the structure inspection process when inspecting a structure, which is the object of inspection, using a photographed image of the structure in the information processing device of this embodiment. In this embodiment, by executing the structure inspection process using the functional blocks of Figure 2, it is possible to reduce the variation in the criteria for determining abnormalities in the structure while also reducing the effort required prior to the inspection work. The structure inspection process using each functional block shown in Figure 2 is realized, for example, by the CPU 101 of Figure 1 executing the program according to this embodiment and controlling each part. Note that the functional blocks shown in Figure 2 may be realized by a hardware configuration, a software configuration, or a combination of a hardware configuration and a software configuration. This also applies to the functional blocks of each embodiment described later.

The target image input unit 201 acquires an image of a structure to be inspected (hereinafter, referred to as a target image) from the NETIF 104 or the HDD 107 .
The target image display unit 202 displays a partial area of the target image input to the target image input unit 201 in a predetermined display area on the screen of the display device 105. The partial area is determined by the user specifying an area in the target image by scrolling, zooming in/out, or the like via the input device 106.

The deformation designation unit 203 designates deformations in a partial area of the target image displayed in the target image display unit 202 based on input from the user via the input device 106. In this embodiment, the deformations are, for example, cracks, lifting, peeling, efflorescence, cold joints, junk (peanut brittle), surface bubbles, sand streaks, rust, etc. that occur on the surface of a structure. In this embodiment, the deformation is designated by the user encircling an area in the target image that is thought to be deformed with a rectangle or ellipse via the input device 106, tracing the outline, etc. If the deformation is linear, such as a crack, the deformation may be designated by tracing over the linear deformation. The deformation designation unit 203 sends information representing the deformation designated in this way as a rectangle surrounding the deformation, the outline of the deformation, or a polyline tracing the deformation itself to the target image display unit 202. As a result, the target image display unit 202 displays the rectangles surrounding the deformations, the contours of the deformations, polylines, etc., on the target image. Since polylines tend to have a large amount of data, the deformation designation unit 203 may calculate the circumscribing rectangle of the polyline and send information expressing the deformation area as a rectangle to the target image display unit 202. The deformations specified by the rectangle in this way are expressed, for example, by the coordinates of the upper left corner and the size (width and height) of the rectangle, which makes it possible to uniquely identify the position and range within the target image. In this embodiment, the deformations are designated by surrounding them with a rectangle as described above. Specific display examples of partial areas of the target image and rectangles that designate deformations within the target image will be described later.

The anomaly determination unit 204 acquires the anomaly determination result input by the user via the input device 106 for the partial area of the target image for which an anomaly has been specified by the anomaly specification unit 203. In this embodiment, the anomaly determination result input by the user includes, for example, the type of anomaly and the degree of the anomaly. Anomaly determination by the user and specific examples of the anomaly determination results will be described later.

The reference information display unit 207 displays the reference information managed by the reference information storage unit 206, which will be described later, on the display device 105. The reference information is information that associates information that indicates at least the type of deformation and the degree of the deformation of a reference image, which is used as a reference when performing a deformation determination of a target image, with the reference image, which is used as a reference when performing a deformation determination of a target image. In this embodiment, the reference image includes a partial image of the target image that includes a deformation specified by the deformation designation unit 203. The reference information includes information that indicates the result of the deformation determination determined by the deformation determination unit 204 in the partial area. The information associated with the reference image may also include display parameters acquired from the reference information storage unit 206 by the display parameter acquisition unit 205, which will be described later, or display parameters acquired from the partial area specified by the deformation designation unit 203. In this embodiment, multiple pieces of reference information are stored, for example, in the HDD 107 or an external recording device via the NETIF 104, and the reference information storage unit 206 manages the multiple reference information stored therein.

The reference information selection unit 209 displays a list of multiple pieces of reference information managed by the reference information storage unit 206 in a specified display area of the display device 105, and allows the user to select one of them. In the list display, reference images are obtained based on the reference information, and thumbnail images created by applying appropriate trimming or reduction processing to the reference images may be displayed side by side, or character strings that can uniquely identify the reference images may be displayed side by side. Specific examples of displaying the list of reference information will be described later.

When displaying a list of reference information, the reference information may be grouped, sorted, or filtered according to the type and degree of the deformation contained in each piece of reference information. In this case, the reference information selection unit 209 has a grouping function for grouping multiple pieces of reference information based on the type and degree of each piece of reference information, a sorting function for sorting based on the type and degree of the reference information, and a filtering function for filtering based on the type and degree of the reference information.

When a piece of reference information is selected from the list display by the user via the input device 106, the reference information display unit 207 acquires a reference image based on the selected reference information and displays it in a predetermined display area of the display device 105. Specific examples of displaying a reference image will be described later.

When abnormality designation and abnormality determination are performed for a partial region of the target image, the information processing device of this embodiment displays the partial region of the target image and the reference image on the display device 105 so that the user can visually compare them. When displaying the partial region of the target image and the reference image so that they can be compared, the information processing device of this embodiment performs an adjustment process to match the appearance (display manner) of the target image and the reference image on display. In the first embodiment, the display manner of the target image is matched to the display manner of the reference image, thereby realizing an adjustment process to match the appearance of the target image and the reference image on display.

For this reason, the display parameter acquisition unit 205 acquires display parameters from a reference image based on the reference information selected by the reference information selection unit 209. Furthermore, if the selected reference information already contains display parameters, the display parameter acquisition unit 205 acquires the display parameters contained in the selected reference information from the reference information storage unit 206. In this embodiment, the display parameters are parameters used to align the display methods of images that look different and to align the appearance on the display.

In this embodiment, by displaying the target image and the reference image in a comparable manner, it becomes easy to compare the abnormality already identified in the reference image with the candidate abnormality before judgment included in the target image, and the work of judging the abnormality in the target image is supported. Therefore, in this embodiment, "matching the appearance on the display" means making the display environment of each of the target image and the reference image equivalent so that it is easier to compare them. For example, the zoom magnification of both is matched. In this embodiment, since the reference image includes a partial image of the target image in which an abnormality has already been judged, by matching the zoom magnification of the target image and the zoom magnification of the reference image, if an abnormality of the same width is shown in each of the target image and the reference image, it appears to be the same width on the display. Details of the display parameters will be described later using a specific example in FIG. 6. Then, the display parameter acquisition unit 205 sends the acquired display parameters to the display parameter application unit 208.

The display parameter application unit 208 applies the display parameters to a partial region of the target image, thereby adjusting the image so that the appearance of the image from which the display parameters are obtained (reference image) matches the appearance of the image to which the display parameters are applied (target image). Specific examples of applying the display parameters will be described later.

In addition, when a partial area of a target image is designated as abnormal and an abnormality determination is performed, the reference information storage unit 206 of this embodiment generates and manages information that associates the abnormality determination result (type and degree of abnormality, etc.) with that partial area as new reference information. The reference information storage unit 206 then stores the newly generated reference information in, for example, the HDD 107 or an external recording device via the NETIF 104. In other words, in this embodiment, the reference information related to the partial area of the target image where an abnormality has been determined by the user can be used as reference information when an abnormality determination is subsequently performed on this target image.

3 is a diagram illustrating a reference information table used to manage the reference information stored by the reference information storage unit 206. In the following, an example of anomalies to be dealt with in a structure inspection, particularly cracks, will be described.
In FIG. 3, a reference ID for uniquely identifying reference information stored in the reference information table is described in a reference ID field 301 of the reference information table.
The relative coordinates of a rectangle representing the region of the abnormality in the target image are described in the coordinate item 302. In this embodiment, the coordinates described in the coordinate item 302 are assumed to be the relative coordinates of the upper left corner of the rectangle in the target image, but are not limited to this and may be the center coordinates or the lower right coordinates of the rectangle.

The size of the rectangle representing the abnormality area is described in the size item 303. Here, the size of the rectangle is made up of the width and height of the rectangle.
A character string representing the type of deformation is described in the type item 304. A value representing the degree of deformation is described in the degree item 305. In the example of this embodiment, the width of the crack is described as the value representing the degree of deformation.
Display parameters are described in the display parameter item 306. As described above, the display parameters are used to match the appearance of another image (a partial region in this embodiment) with respect to the reference image specified by the coordinates in the coordinate item 302 and the size in the size item 303. In this embodiment, a case where a zoom magnification is adopted as an example of the display parameters will be described.

4(a) and 4(b) are diagrams for explaining an example of a display on the display device 105 when a structure inspection process is being executed in the information processing device of this embodiment. The difference between Fig. 4(a) and Fig. 4(b) is that the method of expressing the reference information list is different.
The target image read button 401 is a button that is operated by the user when the target image is to be read by the target image input unit 201 .

The target image display area 402 is a display area in which the target image is displayed by the target image display unit 202. After acquiring one target image, the user can scroll or zoom in and out to move the partial area displayed in the display area 402 while determining the abnormality and inspecting the required range of the target image. For example, scroll bars 409 and 410 can be used to move the partial area. In addition, in response to a specific operation on the target image display area 402 (e.g. right-click, double-tap, etc.),
An interface for changing display parameters such as zoom magnification, brightness adjustment value, hue adjustment value, gamma value, etc. may be displayed.

The reference image display area 403 is a display area in which the reference image is displayed by the reference information display unit 207 .
The deformation area indication rectangle 404 is a rectangle that represents the deformation area specified by the deformation specification unit 203 .
The deformation determination button 405 is a button that is operated when the user inputs the type and degree of deformation by the deformation determination unit 204 for the deformation indicated by the deformation area indication rectangle 404. Although not shown in the figure, when the deformation determination button 405 is pressed, the information processing device displays a dialog window or the like on the screen for inputting the type and degree of deformation, and prompts the user to input them.

4A is an area that displays a list of reference information managed by the reference information storage unit 206. This allows the user to select reference information from the list.
The reference thumbnail image 407 is a thumbnail image representing a reference image, and is created by appropriately trimming, reducing, or enlarging the reference image acquired based on the reference information. In the example of Fig. 4(a), when a deformation is determined, a partial image corresponding to the deformation area indication rectangle 404 designated by the deformation designation unit 203 is trimmed, and the reduced thumbnail image is displayed in the reference information list area 406.
Figure 4 (a) shows an example in which reference thumbnail images 407 are grouped based on the type and degree of deformation contained in the reference information (in this example, the crack width of the crack) and listed in the reference information list area 406.

The reference information list area 408 in Fig. 4(b) is an area where a list of reference information managed by the reference information storage unit 206 is displayed. The reference information list area 406 illustrated in Fig. 4(a) displays a list of thumbnail images representing reference images, while the reference information list area 408 in Fig. 4(b) displays a list of character strings that can uniquely identify the reference images. An example of a character string that can uniquely identify a reference image is a character string such as "deformation_(reference ID)".
Figure 4 (b) shows an example in which character strings that can uniquely identify reference images are grouped based on the type and degree (crack width) contained in the reference information, and are displayed in a list in the reference information list area 408.

In Fig. 4(a) and Fig. 4(b), an example is shown in which the target image display area 402 and the reference image display area 403 are displayed side-by-side so that they can be compared, but a display such as that shown in Fig. 5 may also be used. Fig. 5 shows a display example in which the target image display area 402 and the reference image display area 403 are arranged so that they are superimposed on each other in the same area on the screen (the same display area), and the visibility of the two is switched so that they can be compared.

In the example of FIG. 5, the inspection/reference image display area 501 is a display area for switching between the target image display area 402 and the reference image display area 403. Switching between the target image display area 402 and the reference image display area 403 in the inspection/reference image display area 501 is performed, for example, by operating a toggle button such as the inspection/reference image switching button 502. Note that in FIG. 5, the target image load button 401, the abnormality determination button 405, the reference information list area 406, and the reference thumbnail image 407 are the same as those in FIG. 4(a).

Hereinafter, a process for aligning the display manner of the target image and the reference image (a process for aligning the appearance) will be described with reference to Fig. 6. Also, a process for generating new reference information based on the result of the abnormality determination by the user will be described with reference to Fig. 7.
Fig. 6 is a flowchart for explaining the process executed when reference information is selected from the reference information list in the information processing device of this embodiment. In this embodiment, the reference information is generated based on the abnormality in the target image displayed on the target image display unit 202, as described later in Fig. 7. The process shown in the flowcharts of Figs. 6 and 7 below is realized by the CPU 101 of Fig. 1 executing the program according to this embodiment. In the following description, steps S601 to S604, step S701, and step S702 are abbreviated as S601 to S604, S701, and S702, respectively. The same applies to other flowcharts described later.

In S<b>601 , when the user selects one of the reference information list areas ( 406 or 408 ) described above, the reference information selection unit 209 acquires, from the reference information storage unit 206 , reference information corresponding to the selection.
Next, in S602, the display parameter acquisition unit 205 acquires display parameters from the reference information acquired in S601. Here, the above-mentioned zoom magnification is acquired as an example of the display parameters.

Next, in S603, the reference information display unit 207 displays a reference image in the reference image display area 403 based on the reference information acquired in S601. At this time, the reference information display unit 207 first acquires the upper left coordinate and size of a rectangle surrounding the deformation from the selected reference information. Then, the reference information display unit 207 cuts out a rectangular image based on the upper left coordinate and size of the rectangle from the image from which the reference information was generated (i.e., the target image displayed by the target image display unit 202). Furthermore, the reference information display unit 207 applies the zoom magnification acquired by the display parameter acquisition unit 205 in S602 to the cut-out rectangular image to reproduce the display manner at the time of the deformation judgment when the reference information was generated. The reproduction of the display manner can be realized by applying a scale transformation matrix created based on the zoom magnification to the cut-out rectangular image. Then, the reference information display unit 207 displays the rectangular image that reproduces the display manner as a reference image in the reference image display area 403.

In this embodiment, if the rectangular image that reproduces the display of the rectangle (inside the deformation area indication rectangle 404) surrounding the deformation when the reference information was generated is smaller than the reference image display area 403, the periphery of the rectangle surrounding the deformation is also displayed. As a result, when the reference thumbnail image 407 is selected, the reference image displayed in the reference image display area 403 is the part of the target image that was displayed in the entire target image display area 402 when the deformation was determined. However, this is not limited to this. For example, the rectangle surrounding the deformation when the reference information was generated may be displayed so as to be located at the center of the reference image display area 403. Also, if the rectangle surrounding the deformation is smaller than the reference image display area 403, the outside of the rectangle may be left blank.

Next, in S604, the display parameter application unit 208 applies the zoom magnification acquired by the display parameter acquisition unit 205 in S602 to the target image displayed on the target image display unit 202, thereby reproducing the same display manner as the reference image. This realizes image adjustment to match the appearance (display parameters) of the target image and the reference image. In this case, as described above, a scale transformation matrix created based on the zoom magnification may be applied to the target image. If the selected reference image is a partial image of the target image, the zoom magnification of the target image and the zoom magnification of the reference image are matched, so that the width of the appearance of the same width in each of the target image and the reference image is matched on the display device. However, in this embodiment, even after the reference image is selected and the display parameters of the target image are changed by the information processing device, the user can move the displayed partial area by scrolling or zooming in/out to change the zoom magnification. For example, after confirming that the target image contains a deformation having a width similar to the referenced deformation by displaying it at the same zoom magnification as the reference image, the length of the deformation can be confirmed by zooming out arbitrarily. Then, when an abnormality is judged after zooming out at an arbitrary angle, the zoom magnification finely adjusted by zooming out is held as reference information for the reference image newly generated in response to the judgment. In the first embodiment, when the display parameters of the target image are changed by the user, the change does not affect the reference image.

In the present embodiment, an example has been given in which the zoom magnifications of the target image and the reference image are aligned to match the appearance of the two images, but display parameters other than the zoom magnification may also be used. In this case, possible display parameters include, for example, a brightness adjustment value, a hue adjustment value, and a gamma value. Furthermore, the display parameters used may not only be any one of the zoom magnification, brightness adjustment value, hue adjustment value, and gamma value, but also a combination of these.

For example, when a brightness adjustment value is used as a display parameter, in S602, the display parameter acquisition unit 205 acquires brightness from the reference image to generate a brightness adjustment value. Then, in S604, the display parameter application unit 208 uses the brightness adjustment value to adjust the brightness of the target image. Also, for example, when the HSV color space is used, the maximum of RGB becomes the brightness adjustment value V, and in this case, the display parameter application unit 208 adjusts the brightness adjustment value Vi of the target image to the brightness adjustment value Vr of the reference image. At this time, it is necessary to change values other than Vi (for example, if R is Vi, G and B) according to the change in the brightness adjustment value Vi of the target image so as not to destroy the ratio of RGB before the change. Regardless of which display parameter is adopted, the environment in which the target image and the display image are displayed can be aligned, making it easier to compare the abnormality already identified in the reference image with the candidate for change before the determination included in the target image.
In this embodiment, the above-described process is performed every time reference information is selected from the reference information list.

FIG. 7 is a flowchart illustrating a process for generating new reference information, which is executed when the abnormality determination button 405 is pressed in the information processing device of this embodiment.
In S701, when the deformation determination button 405 is pressed, the deformation determination unit 204 generates new reference information including the partial area of the target image determined to be deformed at that time and information indicating the type and degree of deformation, and stores the information in the reference information storage unit 206. In this case, the reference information storage unit 206 describes the reference information in each item of the reference information table of Fig. 3 described above. The reference information at this time also includes the current zoom magnification of the target image acquired by the display parameter acquisition unit 205.
Depending on the type of display parameter, it is not always necessary to acquire and store the display parameter at this timing. For example, in the case of the brightness adjustment value described above as an example of the display parameter, the brightness adjustment value may be calculated when the reference information is generated and stored as part of the reference information, or the brightness adjustment value may be calculated when the reference image is selected.

Next, in S702, the reference information display unit 207 additionally displays the display element generated based on the reference information generated in S701, for example, in the reference information list area 406 in Fig. 4(a). Specifically, the reference information display unit 207 resizes the deformed part of the reference image generated based on the new reference information, for example, to the size of a thumbnail image, and additionally displays it in the reference information list area 406. Alternatively, the reference information display unit 207 may generate an appropriate character string based on a reference ID indicating the reference information, and additionally display it, for example, in the reference information list area 408 in Fig. 4(b).
The flowchart in FIG. 7 may be started not only when the abnormality determination button 405 is pressed, but also at any other timing, such as when the user specifies an abnormality.

Here, with reference to FIG. 14, an example of transition of the display state on the display device 105 when a structure inspection process is executed in the information processing device of this embodiment will be described.
14(a) to (d) show, in chronological order, the changes in the display state that occur when a user performs a structure inspection operation using the screen shown in FIG. 4(a).
In FIG. 14A, the user sets an abnormality designation area 404 in the target image display area 402 as shown in FIG. 4A.

Here, thumbnail image 1400 is selected as the reference image, and as a result, a reference image based on thumbnail image 1400 is displayed in reference image display area 403. In FIG. 14(a), the selected thumbnail image 1400 is outlined in a thick border, clearly indicating that it can be distinguished from other thumbnail images. Furthermore, in FIG. 14(a), the user operates determination button 405, and the inside of deformation area indication rectangle 404 is registered as a crack with a width of 5.0 mm. As a result, thumbnail image 1401 is added to reference information list area 406.

14B shows a state in which the user, in proceeding with the work, selects a thumbnail image 1402 grouped as a 0.2 mm-wide defect from the reference information list area 406. The selected thumbnail image 1402 is outlined in a thick frame, making it clearly identifiable from other thumbnail images.
In the reference image display area 403, a reference image based on the selected thumbnail image 1402 is displayed.

Here, the area cut out as thumbnail image 1402 from the entire reference image displayed in reference image display area 403 is the lower right portion. In response to the change in the reference image, the zoom magnification of the target image is changed in target image display area 402, and the target image is enlarged more than in the state shown in FIG. 14(a).

The target image displayed in FIG. 14(b) is the part of the target image displayed in FIG. 14(a) that is below the part selected in the abnormality designation area 404. For example, when the user wants to focus on an abnormality with a narrower width compared to the state in FIG. 14(a) (where the 0.5 mm-wide abnormality was referenced), the user selects thumbnail image 1402 as shown in FIG. 14(b) to compare it with the 0.2 mm-wide abnormality. In response to the selection of thumbnail image 1402, the information processing device zooms in on the target image. As a result, the user can easily compare the newly focused fine abnormality in the target image with the 0.2 mm-wide abnormality at the same zoom magnification.

Figure 14 (c) shows the state in which the user has set a new abnormality designation area 1403 in the target image display area 402, pressed the judge button 405, and registered an abnormality with a width of 0.2 mm. As a result, the thumbnail image 1403 is added to the 0.2 mm wide grouping in the reference information list area 406. The thumbnail image 1404 is a reduced image in which the portion of the target image surrounded by the abnormality designation area 1403 has been cut out.

In this manner, in this embodiment, each time a thumbnail image is selected from the reference information list area 406, the display parameters (e.g., zoom magnification) in the target image display area 402 are changed. Also, each time a new abnormality is determined in the target image, reference information is generated. At that time, in the display state, a character string that can uniquely identify the reference information is added to the reference information list area 406.

Note that Figures 14(b) and 14(c) illustrate the task of determining a 0.2 mm wide anomaly in a target image while referring to a reference image of a 0.2 mm wide anomaly, but it is not necessary to refer to an anomaly of the same width. For example, when compared with a reference image of a 0.5 mm wide anomaly, it may be confirmed that the anomaly in the target image is narrower than that in the reference image, and as a result, a 0.2 mm wide anomaly may be determined. In other words, when a new thumbnail image is added to the reference information list, it is not necessarily added to the same group as a thumbnail image that is currently selected.

The newly generated reference information in this embodiment can be selected for reference in a subsequent abnormality determination operation.
For example, FIG. 14D shows a state in which the newly added thumbnail image 1401 in FIG. 14A has been selected.
In the reference image display area 403, a partial image of the target image that was displayed when the abnormality determination was performed on the abnormality designated area 404 in Fig. 14(a) is displayed as a reference image. Also, at this time, in the target image display area 402, the target image is displayed at the zoom magnification of the target image that was displayed when the abnormality determination was performed on the abnormality designated area 404 in Fig. 14(a). As a result, in Fig. 14(a) and Fig. 14(d), the images displayed in the target image display area 402 and the reference image display area 403 look exactly the same.

However, this is not limited to the present embodiment. For example, in the reference image display area 403, a reference image may be displayed such that the area cut out in the selected thumbnail image 1401 is at its center. Also, for example, in the target image display area 402, the displayed portion may be aligned so that the partial image that was displayed immediately before the zoom magnification was changed is at its center.

In this way, in the first embodiment, a new reference image can be generated from the target image based on the user's anomaly judgment result. As a result, according to the first embodiment, it is possible to eliminate the need to prepare reference images in advance for each type and degree of anomaly. In addition, in the first embodiment, the conditions (environment) for how the target image and the previously judged anomaly image (reference image) are viewed can be aligned. Specifically, in order to display the target image and the reference image, which is a part of the target image, in a state that makes it easier to compare them, the display parameters of each are matched. As a result, according to the first embodiment, it becomes possible to judge new anomalies by referring to anomalies in previously judged anomaly images (reference images), making it possible to provide consistency in the anomaly judgment.

Second Embodiment
In the first embodiment described above, the conditions for how the target image appears are aligned based on the reference information selected by the user. Also, when the user performs an operation to change the appearance of the target image, the effect is not extended to the reference image. In contrast, in the second embodiment described below, an operation is described in which, when the way the target image is displayed is changed by a user operation such as changing the zoom magnification of the target image display area 402, the change is reflected in the display mode of the reference image. That is, in the second embodiment, a process of adjusting display parameters is performed in order to align the conditions (environment) for how the reference image appears based on the target image.

Figure 8 is a diagram showing the functional blocks for realizing structure inspection processing in an information processing device of the second embodiment. The target image input unit 201 to the deformation determination unit 204 and the reference information storage unit 206 to the reference information selection unit 209 are similar to the corresponding units in Figure 2, so their description will be omitted. The display parameter acquisition unit 801 in the second embodiment acquires display parameters from the target image or the target image display unit 202.

Furthermore, Fig. 9(a) and Fig. 9(b) are flowcharts showing two types of processing executed when the way in which the target image is displayed is changed by a user operation in the structure inspection processing in the information processing device of the second embodiment. The processing of the flowcharts in Fig. 9(a) and Fig. 9(b) below is started when the parameters of the target image display area 402 are changed by the user.

The flowchart in FIG. 9A shows a process for applying display parameters changed in a target image as display parameters of one selected reference image to match the appearance of the two images.
In S901, the display parameter acquisition unit 801 acquires the current display parameters set in the target image display unit 202. In the second embodiment, the display parameters are, for example, a zoom magnification, a brightness adjustment value, a hue adjustment value, a gamma value, and the like.
Next, in S902, the display parameter application unit 208 changes the zoom magnification of the reference image displayed on the reference information display unit 207 to the zoom magnification acquired in S901.

Now, referring to FIG. 15, an example of the transition of the display state on the display device 105 when the process of the flowchart in FIG. 9(a) is executed will be described. FIGS. 15(a)-(b) show, in chronological order, the changes in the display state that occur when the process of the flowchart in FIG. 9(a) is executed when a user performs structure inspection work in the second embodiment.

FIG. 15(a) shows the same state as FIG. 14(b).
In the reference image display area 403, a reference image based on a selected thumbnail image 1402 is displayed.
15A, a zoom magnification change button 1501 is displayed in the target image display area 402. The zoom magnification change button 1501 is an example of an interface for operating display parameters that can be called up in response to a predetermined operation, such as a click or a double tap, in the target image display area 402. By operating the zoom magnification change button 1501, the user can arbitrarily change the zoom magnification of the target image.

15B shows a state in which the inside of the target image display area 402 is further zoomed in in response to a user operation. In this embodiment, in response to a user operation on the target image, the same change in display parameters as the target image is reflected in the reference image display area 403.
In the case of Fig. 15(b), the reference image is also zoomed in. In this embodiment, particularly when the reference image is zoomed in (enlarged), the partial area is aligned so that the already determined abnormality is always included in the displayed area so that the abnormality in the reference image is not lost. As a result, the abnormality cut out in the thumbnail image 1402 is enlarged and displayed in the reference image display area 403 in Fig. 15(b).

The flowchart in Fig. 9B shows a process of applying the display parameters changed in the target image as display parameters reflected not only in the selected reference image but also in all reference information included in the display information list. However, in the explanation of Fig. 9B, the reference information list is composed of thumbnail images of reference images obtained based on each reference information, which are reduced or enlarged without being cropped.
The process of S901 in FIG. 9B is similar to the process of S901 in FIG. 9A, and therefore description thereof will be omitted.

Next, in S903, the reference information selection unit 209 determines whether the subsequent steps S904 and S905 have been completed for all reference information stored in the reference information storage unit 206. If the reference information selection unit 209 determines in S903 that the processing has been completed, it ends the processing of the flowchart in Fig. 9B. On the other hand, if the reference information selection unit 209 determines in S903 that the processing has not been completed, it proceeds to S904.
In S904, the reference information selection unit 209 acquires, from the reference information storage unit 206, one piece of reference information for which the subsequent processes in S904 and S905 have not been completed.

Next, in S905, the reference information selection unit 209 obtains a reference image based on the reference information obtained in S904 using the method described in S603. The reference information selection unit 209 then applies a scale transformation matrix created based on the zoom factor obtained in S901 to the reference image, and then resizes the reference image to the size of a thumbnail image. However, in this method, resizing processing must be performed twice, which increases the amount of calculation. For this reason, it is possible to combine the zoom factor obtained in S901 with a scale coefficient for thumbnail generation calculated from the size ratio between the reference image and the thumbnail image, so that a single resizing process is sufficient.
Note that, after the process of the flowchart in FIG. 9B is completed, when a thumbnail image is selected from the reference information list, the process of the flowchart in FIG. 9A may be executed.

Here, with reference to FIGS. 15C and 15D, an example of transition of the display state on the display device 105 when the process of the flowchart in FIG. 9B is executed will be described.
Figures 15 (c) to (d) show in chronological order the changes in the display state that occur when the processing of the flowchart in Figure 9 (a) is executed when a user performs structure inspection work in the second embodiment.

In FIG. 15(c), the state of the target image display area 402 is the same as in FIG. 15(a). In the reference image display area 403, a reference image based on the selected thumbnail image 1503 is displayed. Here, the reference image corresponding to the thumbnail image 1503 is the same as the thumbnail image 1402 in FIG. 15(a). However, in FIG. 15(c), the thumbnail image displayed in the reference information list area 406 is a reduced version of the reference image itself. In this case as well, the user can change the zoom magnification of the target image as desired by operating the zoom magnification change button 1501.

Figure 15(d) shows a state in which the inside of the target image display area 402 has been further zoomed in in response to a user operation. In this embodiment, in response to a user operation on the target image, each of the thumbnail images displayed in the reference information list area 406 is zoomed in. In the case of Figure 15(d), the reference image is also zoomed in by further performing the process of Figure 9(a).

In this way, according to the information processing device of the second embodiment, the appearance of the reference image can always be aligned with the target image based on the user's operation on the target image, thereby enabling the user to quickly determine any abnormalities.

Third Embodiment
In the first embodiment described above, a reference image was generated from the target image being displayed. In contrast, in the third embodiment, previously generated reference information is read (including reading from outside). In this case, it is originally recommended to shoot the deformed part at a predetermined resolution, but this is not always met depending on the situation at the shooting site. When previously generated reference information is read as in the third embodiment, the resolution of the reference image based on the read reference information may differ from the resolution of the captured target image. If the resolutions of the target image and the reference image are different in this way, it is difficult to align the display methods of both images even if the zoom magnification is simply adjusted as described above.

10 is a functional block diagram for implementing the structure inspection process in the information processing device of the third embodiment. The target image display unit 202 to the reference information selection unit 209 are similar to the corresponding units in FIG. 2, and therefore the description thereof will be omitted.
10, a target image input unit 1001 acquires a target image from the HDD 107 or the NETIF 104. At that time, the target image input unit 1001 requests the resolution (unit: pixels/mm) of the target image to be input from the user. The resolution of the target image input by the user is held as temporary data (e.g., held in a predetermined area of the RAM 102) while the target image is displayed on the target image display unit 202.

The reference information input unit 1002 acquires reference information from the HDD 107 or the NETIF 104. In this case, the reference information may be saved as a file or stored in a database. In order to input appropriate reference information for the target image to be inspected, the multiple pieces of reference information may be structured so as to be searchable. In this case, the reference information input unit 1002 has a search function for searching by the type and level of reference information, for example. In this case, the type and material of the structure where the abnormality occurred, the position where the abnormality occurred, the person who made the judgment, etc. can be used as search keys. The reference information acquired by the reference information input unit 1002 is temporarily stored, for example, by the reference information storage unit 206.

The reference information output unit 1003 outputs the reference information newly generated in the information processing device of the third embodiment to the HDD 107 or the NETIF 104. The output reference information may be saved as a file or stored in a database.

11 is a diagram illustrating a reference information table managed by the reference information storage unit 206 in the third embodiment. The reference ID item 301 to the display parameter item 306 are the same as the corresponding items in FIG. 3, and therefore the description thereof will be omitted.
The reference information table in FIG. 11 includes an image path item 1101 and a resolution item 1102 .
The image path item 1101 describes a path indicating an image in which the referenced defect appears.
The resolution item 1102 describes the resolution of the image indicated by the path in the image path item 1101 (the image in which the referenced defect is reflected).

12 is a diagram illustrating an example of a display screen in the information processing apparatus of the third embodiment. The target image load button 401 to the reference thumbnail image 407 are similar to the corresponding display elements in FIG. 4, and therefore the description thereof will be omitted.
The reference information input button 1201 is a button operated by the user when inputting reference information through the reference information input unit 1002 .
The reference information output button 1202 is a button that is operated by the user when the reference information output unit 1003 outputs the reference information.

Figure 13 is a flowchart explaining the processing executed when reference information is selected from a list of previously generated reference information in the structure inspection processing realized by the information processing device of this embodiment. S601 is the same as the corresponding step in Figure 6, so its explanation is omitted. In the case of the flowchart of Figure 13, after S601, the processing proceeds to S1301.

In S1301, the reference information selection unit 209 acquires the path and resolution of the image in which the abnormality is reflected, and the zoom magnification at the time of abnormality determination, from the reference information selected by the user.
Next, in step S1302, the target image display unit 202 acquires the resolution of the target image being displayed. The resolution of the target image is assumed to be stored as temporary data associated with the target image being displayed on the target image display unit 202.

Next, in S1303, the target image display unit 202 matches the appearance (apparent resolution) on the display of the target image being displayed with the image (the reference image identified from the images showing the defect) whose path, resolution, and zoom magnification were obtained in S1301. Specifically, the target image display unit 202 applies a scale coefficient calculated using equation (1) described below to the one of the two images with the higher resolution.

Here, the scale coefficient _S1 of the image size is calculated as shown in formula (1) where the resolution acquired in S1301 is _resa and the resolution acquired in S1302 is _resb .

In this example, the image with the higher resolution is appropriately scaled to match the appearance (apparent resolution) of the other image on the display, but the image with the lower resolution may also be appropriately scaled to match the apparent resolution of the other image on the display. In this case, the scale coefficient _S2 calculated by the following formula (2) may be applied to the image with the lower resolution.

Next, in S1304, the display parameter application unit 208 applies the zoom magnification obtained in S1301 to the target image displayed on the target image display unit 202 after matching the appearance (apparent resolution) of both images in S1303.

In this way, in the third embodiment, it is possible to reuse reference information created when another user performs inspection work using the information processing device of this embodiment. As a result, according to the third embodiment, reference information created by a user (expert) who is skilled in inspection work can be used by an inexperienced user (novice), and even a beginner can make a judgment that matches the judgment criteria of the expert.

In the third embodiment, the display method of the target image is aligned with the reference image, but of course the display method of the reference image may be aligned with the target image. The processing in this case is basically as described in the flowchart of FIG. 9(a) or FIG. 9(b), but it is necessary to align the apparent resolution of both images in advance as in S1303. Also, in the third embodiment, an example was given in which previously generated reference information was displayed in a list for selection, but the reference information displayed in a list may include both previously generated reference information and reference information newly generated from the target image as in the previous embodiment.

As described above, according to each of the above-described embodiments of the present invention, it is possible to automatically generate a reference image to be referred to when making an abnormality determination based on the user's abnormality determination result. Furthermore, according to each of the embodiments, it is possible to easily compare the image of the inspection target and the reference image for abnormality determination by aligning their appearances.

<Other embodiments>
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted in a limiting manner based on them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics.

101: CPU, 102: RAM, 103: ROM, 104: NETIF, 105: display device, 106: input device, 402: target image display area, 403: reference image display area, 406, 408: reference information list area

Claims (8)

An acquisition means for acquiring a first captured image of a structure;
a display control means for classifying the second captured images of the structure according to the width of the deformation of the structure and displaying the classified images on the display unit;
a selection means for selecting one of the second captured images based on a user operation;
A determination means for determining a width of the deformation of the structure included in the first captured image based on the second captured image selected by the selection means,
The display control means displays the first captured image and the second captured image selected by the selection means on the same screen , and when a zoom magnification is changed based on a user operation, adjusts the ratio of pixel units to distance units between the first captured image and the second captured image selected by the selection means, and displays the first captured image by aligning its position so that a partial image that was displayed immediately before the zoom magnification was changed is at the center, and displays the second captured image by aligning its position so that the abnormality is included in the displayed partial area.
23. An information processing apparatus comprising: The information processing device according to claim 1, characterized in that the display control means classifies a specific type of deformation according to the width of the deformation of the structure and displays it on the display unit. The information processing device according to claim 1, further comprising a presentation means for grouping and presenting the second captured images for one of the types of deformation based on the width of the deformation. The information processing device according to claim 1, characterized in that the second captured image is generated by trimming an area including a deformation of the structure from an image of the structure. The information processing device according to claim 1, characterized in that the display control means switches between displaying the first captured image and the second captured image selected by the selection means on the same screen. The information processing device according to claim 1, characterized in that the display control means displays the first captured image and the second captured image selected by the selection means side by side on the same screen. An acquisition step of acquiring a first captured image of the structure;
a display control step of classifying the second captured images of the structure according to the width of the deformation of the structure and displaying the second captured images on a display unit;
a selection step of selecting one of the second captured images based on a user operation;
A determination step of determining a width of the deformation of the structure captured in the first captured image based on the second captured image selected in the selection step,
In the display control step, the first captured image and the second captured image selected in the selection step are displayed on the same screen , and when a zoom magnification is changed based on a user operation, the ratio of pixel units to distance units of the first captured image and the second captured image selected in the selection step are adjusted, and the first captured image is displayed with a position adjusted so that a partial image displayed immediately before the zoom magnification was changed is at the center, and the second captured image is displayed with a position adjusted so that the abnormality is included in the displayed partial area.
23. An information processing method comprising: An acquisition step of acquiring a first captured image of the structure;
a display control step of classifying the second captured images of the structure according to the width of the deformation of the structure and displaying the second captured images on a display unit;
a selection step of selecting one of the second captured images based on a user operation;
A determination step of determining a width of the deformation of the structure captured in the first captured image based on the second captured image selected in the selection step,
In the display control step, the first captured image and the second captured image selected in the selection step are displayed on the same screen , and when a zoom magnification is changed based on a user operation, the ratio of pixel units to distance units of the first captured image and the second captured image selected in the selection step are adjusted, and the first captured image is displayed with a position adjusted so that a partial image displayed immediately before the zoom magnification was changed is at the center, and the second captured image is displayed with a position adjusted so that the abnormality is included in the displayed partial area.
A program for causing a computer to execute an information processing method comprising the steps of:

Images