JP7650472B2 - System for inspecting objects and generating reports - Google Patents

Description

The present invention relates to a computer-based system for assisting in the task of examining, inspecting, or evaluating an object using images and creating a report describing the results.

For the purposes of investigating and inspecting man-made facilities and natural objects, the land or facility in question is photographed with a camera mounted on a mobile vehicle in the air or on the ground, or carried by a person, and the person inspects or evaluates the object while looking at the many images obtained. A report expressing the results using a combination of the photographed images and accompanying marks and text comments is prepared in a variety of industrial fields.

Systems for creating reports are disclosed, for example, in Patent Document 1 and Patent Document 2.

JP 2005-352848 A JP 2007-34453 A

For example, when inspecting the condition of a vast object such as a solar power generation facility based on images taken with a camera mounted on an unmanned aerial vehicle, the number of images taken of the entire object will be very large. It is not an easy task for a human to look at these many images to get a complete picture of the object or to inspect each part in detail.

Furthermore, in order to create a report that allows others to easily understand the inspection results, it is necessary to skillfully arrange many images on the report and add marks and comments to the images. This takes a lot of time to create a report. For example, it is said that it usually takes about a week to inspect a solar power generation facility based on images taken from an unmanned aerial vehicle and create a report.

However, the prior art disclosed in Patent Documents 1 and 2 does not address the above problems.

One objective is to provide a computer-based system that assists in the examination of an object using a large number of images obtained from the object, and in the compilation of the results of that examination into a report.

According to one embodiment, a system for inspecting an object and creating a report includes a means for communicating with a user terminal, a means for preparing a plurality of images each representing the entire object and different parts of the object, a means for displaying a work screen on the user terminal that displays an image and a comment input tool requested by a user, a means for receiving a comment entered by the user on the work screen and storing the comment in association with the image displayed on the work screen, a means for storing a report form, and a means for automatically creating a report according to the report form using the stored comment and the image associated therewith.

1 shows an overall structure of an examination report support system according to an embodiment. 2 shows a basic flow of communication between a data processing server and a user in the system. 4 shows the flow of inspection processing performed by the data processing server. 13 shows an example of a screen display that allows a user to get an overview of the entire image of an object to be inspected during inspection processing. 1 shows an example screen display for allowing a user to segment an object to be inspected into sub-regions during the inspection process. 13 shows an example of a screen display for allowing a user to closely inspect a portion of an inspection target during inspection processing. 13 shows an example of a screen display that allows a user to view points of interest in an inspection process and check the inspection results of each point. 13 shows a flow of a comparison inspection process performed by the data processing server. 1 shows an example of a screen display for allowing a user to perform an inspection by comparatively observing two images of an object during inspection processing. 4 shows the flow of a report creation process performed by a data processing server. 1 shows a portion of an example report automatically generated by the data processing server.

Figure 1 shows the overall structure of an examination report support system according to one embodiment.

As shown in FIG. 1, the inspection report support system 1 has a data processing server 3, which may be composed of one or more server computer machines. The data processing server 3 provides a service to one or more users to assist them in inspecting one or more objects and compiling the inspection results into a report. The data processing server 3 can communicate data with a web browser on an information processing terminal (hereinafter, user terminal) 7, such as a personal computer or smartphone used by each user, via a web server 5. For convenience, only one user terminal 7 is shown in FIG. 1, and the following describes the details of this system using an example in which one user uses the data processing server 3.

The data processing server 3 has processing components such as a processing request receiving unit 11, a data storage and search unit 13, an image registration unit 15, an image combination unit 17, a map and image display unit 19, a comment registration unit 21, a report creation unit 23, a report display unit 25, a report editing unit 27, and a report output unit 29.

The processing request receiving unit 11 receives various processing requests from users (for example, various processing requests for performing an inspection, or various processing requests for creating a report, etc.) and passes the request to a processing component that performs the requested processing.

The data storage and search unit 13 stores data such as maps, images, inspection results, and reports of various inspection subjects, and in response to a data request from another processing component, reads the requested map, image, or inspection result data and passes it to that processing component.

The image registration unit 15 receives images of the inspection object from the user terminal 7 (which may include not only typical visible light photographic images, but also photographic images using invisible light such as infrared light, and other types of images), and registers these images in the data storage and search unit 13 in association with the inspection object and its user. Typically, multiple images are registered for one inspection object. For example, when a photograph of a vast object such as a solar power generation facility is taken with a camera mounted on an unmanned aerial vehicle, multiple high-quality photographic images of various parts of the facility are registered in association with the solar power generation facility.

The image combining unit 17 combines multiple images of each registered inspection object to generate various images (hereafter referred to as inspection images) that the user will observe to inspect the object. The inspection images are stored in the data storage and search unit 13.

Images with multiple different resolutions, from the lowest to the highest, may be created as inspection images of the object. For example, relatively low-resolution images for an overview of the entire object, relatively high-resolution images for detailed inspection of individual parts of the object, and images with intermediate resolutions may be created as inspection images. The format of the inspection images and their provision to the user may be determined, for example, in accordance with the Web Map Tile Service (WMTS) or Web Map Service (WMS). The inspection images may be orthoimages or mosaic images. The orthoimage inspection images may be created using Structure from Motion (SfM).

In response to a processing request from a user, the map/image display unit 19 sends a map or an inspection image of the object to be inspected to be displayed on the screen of the user terminal 7 to the Web server 5. The Web server 5 sends a work screen incorporating the map or inspection image to the user terminal 7 and displays it on the screen of the web browser.

The comment registration unit 21 receives comments (text, options, marks, figures, etc.) entered into the user terminal 7 by the user when inspecting an object, and stores them in the data storage and search unit 13 as inspection result data associated with the object.

In response to a user request, the report creation unit 23 automatically creates an inspection report for each inspection object using data such as maps, inspection images, and inspection results for each inspection object stored in the data storage and search unit 13. The automatically created report is associated with the object and stored in the data storage and search unit 13.

In response to a request from a user, the report display unit 25 sends the automatically created inspection report to the Web server 5. The Web server 5 sends the report to the user terminal 7 and displays it on the screen of the web browser.

The report editing unit 27 allows the user to freely edit the report displayed on the screen of the user terminal 7. The report edited by the user is associated with its subject and stored in the data storage and search unit 13.

The report output unit 29 sends the edited report to the Web server 5 in response to a request from the user. The Web server 5 enables the user to download the report to the user terminal 7.

Figure 2 shows the basic flow of communication between the data processing server 3 and the user 9 in the examination report support system 1.

In FIG. 2, in step 31, a user 9 registers images (usually multiple images) of an inspection object in the data processing server 3. In step 33, the data processing server 3 stores the registered images, and in step 35 combines the stored images to create inspection images of the inspection object with different resolutions.

In step 37, the user requests a desired inspection image (e.g., an image for an overview of the entire object, or an image for observing a specific part of the object in detail) to inspect the object. In response to the request, in step 39, the data processing server 3 selects and reads the requested inspection image from among the stored inspection images, and in step 41 causes the inspection image to be displayed on the user terminal 7.

In step 43, the user looks at and examines the displayed inspection image, makes the necessary judgment about the inspection object, and in step 45 inputs comments, etc. according to the judgment result (text, options, graphic marks, etc. that describe the judgment result or opinion). Then, in step 47, the data processing server 3 stores the examined image and the input comments, etc. in association with each other.

In the inspection process for one inspection object, steps 37 to 47 may be repeated multiple times. For example, the user may first take an overview of the object as a whole and input comments, etc., then look at detailed images of each part of the object, inspect those parts, and input comments, etc., repeating this process until all parts have been inspected. Steps 37 to 47 are performed each time this process is repeated.

Then, in step 49, the user requests a report summarizing the test results for that test subject. In response to the request, in step 51, the data processing server 3 automatically creates a report using the stored test results, and in step 53, displays the report on the user terminal 7.

In step 55, the user reviews the automatically generated report, and in step 57, makes any desired edits to the automatically generated report and sends the edited report to the data processing server 3. In step 59, the data processing server 3 stores the edited report, and in step 61, outputs the report. In step 63, the user downloads the output report.

Figure 3 shows the flow of the inspection process (processing corresponding to steps 39, 41, and 47 in Figure 2) performed by the data processing server 3.

As shown in FIG. 3, when the data processing server 3 starts the inspection process, it performs the processes of steps S1, S2, and S3 in parallel. In step S1, it is determined whether an image display request has been made by the user, and what area of the image the user has requested to display, such as the entire inspection object or a specific part, is determined. In step S2, it is determined whether the user has requested to zoom or scroll the displayed image. In step S3, it is determined whether the image is of an area selected by the user within the displayed image.

When it is determined in step S1 that a request has been made to display an image of a certain area of a certain inspection object, the data processing server 3 in step S4 prepares an image of that area of the inspection object and displays a work screen incorporating that image on the screen of the user terminal 7. For example, if the user requests the display of an image that provides an overview of the entire inspection object, a work screen such as that shown in FIG. 4 is displayed, incorporating an image 71 showing a wide area that encompasses the entire inspection object. Alternatively, if the user requests an image for examining a specific part of the inspection object in detail, a work screen such as that shown in FIG. 6 is displayed, incorporating an image 91 showing a narrow area of that specific part.

Referring again to FIG. 3, if it is determined in step S2 that zooming (enlarging or reducing) or scrolling of the display image has been requested, the data processing server 3 performs the requested zooming or scrolling of the display image in step S5. For example, in the work screen illustrated in FIG. 4, the user can zoom in on an image 71 showing the entire object to view a portion of the object in greater detail and at a higher resolution. Or, for example, in the work screen illustrated in FIG. 6, the user can scroll an image 91 showing a particular portion of the object to view another portion of the object.

Referring again to FIG. 3, if it is determined in step S3 that a certain area in the image displayed on the work screen has been selected by the user, the data processing server 3 displays the outline of the selected area on the image in step S6. Furthermore, in step S7, the data processing server 3 displays on the work screen via the Web server 5 one or more tools (hereinafter referred to as comment input tools) for the user to input the area name of the selected area, comments on the inspection results, etc. For example, on the work screen shown in FIG. 4, the outline 75 of the area selected by the user (in this example, the selected area is the entire site of the solar power generation facility to be inspected) is displayed on the image, and comment input tools 76, 77 for inputting the area name and any comments regarding the selected area are displayed.

Referring again to FIG. 3, when the user uses the displayed comment input tool to input the area name of the area or a comment related to the area, the data processing server 3 displays the input area name or comment on the work screen in step S8. For example, in the work screen illustrated in FIG. 4, the input area name is displayed in the comment input tool 76, and the input comment is displayed in the comment display field 79 together with the timestamp of the input.

Referring again to FIG. 3, in step S9, the data processing server stores the image displayed on the work screen, the position coordinates of the area (or area frame) selected by the user, and the area name and comments entered by the user as data elements constituting the inspection results of the selected area, in association with the selected area.

Figures 4 to 7 show some examples of work screens for inspecting an object, which are displayed on the screen of the user terminal 7 by the inspection process shown in Figure 3. In the examples shown in Figures 4 to 7, a solar power generation facility is the object of inspection.

Figure 4 shows an example of a work screen that allows the user to get an overview of the entire solar power generation facility to be inspected and select the entire area of the site.

As shown in FIG. 4, image 71 displayed on the work screen is a relatively low-resolution image showing the entire solar power plant. The user can zoom in on the displayed image to display a higher resolution image and view the solar power plant in detail.

The user can use an area selection tool 73 on the display image 71 to select the object to be inspected, i.e., the entire site of the solar power generation facility (typically by drawing the outline of the entire site). Then, an outline frame 75 of the selected area is displayed on the image 71. At the same time, tools 76, 77 for inputting an area name and comments about the selected area (comment input tools) are displayed on the work screen.

The user can use input tool 76 to input the name of the selected area (for example, "A Prefecture, B City, C Town, E Facility"), and can use input tool 77 to input any comments, such as the inspection results and explanations, for the selected area. When the user clicks the send button of the input tools 76 and 77, the input area name and comment are sent to the data processing server 3 and stored in association with the display image 71 and the position coordinates of the selected area (or area frame). In the example of FIG. 4, in addition to the comment input tools 76 and 77, any comments previously input and saved about the selected area are also displayed in the comment display tool 79.

When the user double-clicks on the outline 75 of the object to be inspected on the work screen as shown in FIG. 4, or specifies the object to be inspected in some other predetermined manner, a work screen is displayed on the screen for inspecting the object while examining in detail the condition of each part of the object.

Figure 5 shows an example of a work screen display that allows you to set one or more sub-areas within any already selected area.

The work screen illustrated in FIG. 5 illustrates a case where multiple sub-areas are set within the area of the entire site of the solar power generation facility selected in the example screen of FIG. 4. That is, when the user uses the area selection tool 73 to draw the outline of each sub-area, the outline frames 81A and 81B drawn are displayed on the display image 75. At the same time, comment input tools 83A, 85A, 83B, and 85B for inputting the area name and comments of each sub-area are also displayed on the work screen. When the user inputs the name of each area (for example, "Area A" and "Area B") using the input tools 83A and 83B, each area name is displayed in association with the outline frames 81A and 81B of each sub-area on the display image 75. In addition, when the user inputs comments regarding each area using the input tools 85A and 85B, the comments are displayed in the input tools 85A and 85B, respectively. In the example of FIG. 5, comments previously input regarding the sub-area named "Area" are also displayed in the comment display field 87A.

The user can further set a new sub-area. The comment input tool for the added sub-area is added below the comment input tool 85B at the bottom position shown in FIG. 5. The user can access any comment input tool by scrolling through the comment input tools on the work screen 75.

When the user double-clicks on the outline 81A, 81B or name of any sub-area on a work screen such as that shown in Figure 5, or specifies any sub-area in any other predetermined manner, a work screen is displayed on the screen for inspecting each part of the specified sub-area in detail.

Figure 6 shows an example of a work screen for performing an inspection while examining each part of the inspection target in detail.

The example shown in FIG. 6 is a work screen for detailed inspection of each part of the sub-area "Area" shown in FIG. 5. This work screen displays a relatively high-resolution image 91 that shows in detail the condition of a portion of Area A. In addition, an auxiliary image 95 is also displayed on the work screen, showing with a frame 94 which indicates which part of the object to be inspected corresponds to the area shown by the displayed image 91.

When the user uses the area selection tool 73 to select a location of interest on the display screen 91 (for example, by drawing a frame around the location), a frame 91 of the location of interest is displayed on the display screen 91. At the same time, a mark 96 indicating where the location of the location of interest in the inspection object is located is displayed on the auxiliary image 95. At the same time, tools 97, 99, 101, 103 for inputting the location number, location information, urgency of maintenance, and comments on the condition evaluation related to the location of interest (comment input tools) are also displayed on the work screen.

The location number may be set automatically, or the user may be able to input any number. The urgency level may be set by selecting a standard level such as "high," "medium," or "low" from a menu. The status evaluation comment may also be set by selecting one of several standard comments from a pull-down menu 105 as shown in FIG. 5, or may be freely entered.

When the location number of the attention area (e.g., "(1)") is set or entered, the location number is displayed on the display screen 91 in association with the frame 93 of the attention area.

Figure 7 shows an example of a work screen that displays a list of areas of interest that have already been inspected within the inspection target, and allows the inspection results of each area of interest to be confirmed.

As shown in FIG. 7, location marks 113 indicating each location of interest are displayed on an image 71 that provides an overview of the entire inspection target. When the user designates any location mark 113 with the cursor 111, an image 115 showing the detailed state of the corresponding location of interest and the inspection results 117 of that location of interest are displayed on the work screen.

Figure 8 shows the flow of the comparison inspection process that can be performed by the data processing server during the above-mentioned inspection process.

Comparison inspection processing is processing that displays multiple images of different inspection objects side by side on the work screen, allowing the user to compare and examine the images to perform an inspection. The multiple images displayed can be, for example, images of the same part of the inspection object taken at different times, or images of the same part taken using different methods (for example, visible light photographic images and infrared photographic images).

As shown in FIG. 8, in step S21, the data processing server 3 displays multiple images requested by the user in parallel on the work screen.

Then, in step S22, when the user requests zooming or scrolling of any of the displayed images, in step S23, the multiple images displayed in parallel on the work screen are synchronously zoomed or scrolled to the same zoom factor or scroll direction and distance.

In addition, in step S24, when the user draws a frame indicating an arbitrary area on any of the displayed images, in step S25, the frame of that area is displayed on the multiple images, and in step S26, a tool for inputting test results and the like for that area (a tool for inputting comments, etc.) is displayed on the work screen.

In addition, in step S27, when the user inputs a comment using the comment input tool, the input comment is displayed on the work screen, and in step S28, the input comment, the coordinates of the area (or area frame), and the multiple images on the work screen are associated with each other and saved as inspection result data.

In step S27, when the user requests to end the test, the comparison test process ends.

Figure 9 shows an example of the work screen display for the comparison inspection process.

On the work screen illustrated in FIG. 9, two images 121, 123 showing the same part of the inspection target are displayed side by side. For example, one image 121 is an infrared photographic image and the other image 123 is a visible light photographic image. The images displayed side by side may be images taken at different times by the same or different sensors.

When the user zooms or scrolls one image (e.g., image 121), both images 121, 123 are simultaneously zoomed or scrolled in the same manner.

When the user uses the area selection tool 73 to draw an area frame on one of the images (e.g., image 121), the area frames 125, 127 are displayed on both images 121, 123. In addition, a tool 129 for inputting comments, etc., related to the area is also displayed.

Figure 10 shows the flow of the report creation process performed by the data processing server 3.

As shown in FIG. 10, in step S31, a previously saved report form (data defining the report structure (e.g., what title items are arranged in what order, and what images and text comments are arranged for each item, and in what layout, etc.)) is read out. Here, the report form can be prepared in several different ways. One way is for the user to define the report structure by programming. Another way is for the user to send an image or document file of an existing printed report (e.g., an image file obtained by scanning the report with an image printer, or a document file of the report created with a word processor, etc.) from the user terminal 7 to the data processing server 3, and the data processing server 3 analyzes the file and automatically creates the report form.

In step S32, uncreated items are extracted from the items to be reported defined in the report form. Then, in step S32, images, area coordinates, area names, comments, etc. corresponding to the components of each uncreated item are read from the stored inspection result data, and in step S34, based on the read data, the image, area name, area frame, comments, etc. are arranged and written in the corresponding item part of the report according to the layout defined in the report form. Steps S33 and S34 are repeated until there are no more uncreated items, and the report is created.

Then, in step S35, the report is saved as a file in a format that can be later edited by the user.

Figure 11 shows a portion of an example report that was automatically generated by the process shown in Figure 10.

As shown in FIG. 11, the automatically created report includes the display image used on the work screen as exemplified in FIG. 4 to FIG. 7, the area frame on the image, the area name, the input comment, etc., in the order and layout defined in the report form. The user can complete the report by downloading such an automatically created report file to the user terminal 7 and editing it as necessary.

Although one embodiment has been described above, the scope of the present invention is not limited to this embodiment. For example, the area frames, area names, input comments, etc. on the images included in the automatically generated report may be automatically created by artificial intelligence, etc., and the present invention may be embodied in various other forms without departing from the spirit of the present invention.

REFERENCE SIGNS LIST 1 Inspection report support system 3 Data processing server 7 User terminal 11 Processing request reception unit 13 Data storage and search unit 15 Image registration unit 17 Image combination unit 19 Map and image display unit 21 Comment registration unit 23 Report creation unit 25 Report display unit 27 Report editing unit 29 Report output unit

Claims (3)

In a system for testing objects and generating reports,
means for communicating with a user terminal used by a user;
means for providing a plurality of images representing the object;
a means for displaying on the user terminal a work screen displaying an image and a comment input tool requested by the user;
a means for receiving a comment input by the user on the work screen and storing the comment in association with the image displayed on the work screen;
means for automatically generating a report using the stored comments and the associated images;
when the user draws an area on any one of the plurality of images displayed on the work screen, the display means displays the area on each of the plurality of images at a position corresponding to the area , and displays the comment input tool for inputting a comment regarding the area;
the saving means saves the images displayed on the work screen, the position coordinates of the areas, and the comments inputted by the comment input tool in association with one another;
the means for automatically creating the report creates the report in which the images, the areas, and the comments are arranged, based on the plurality of images, the position coordinates, and the comments that are stored in association with each other;
A system configured as follows. In a system for testing objects and generating reports,
means for communicating with a user terminal used by a user;
means for providing a plurality of images representing the object;
a means for displaying on the user terminal a work screen displaying an image and a comment input tool requested by the user;
a means for receiving a comment input by the user on the work screen and storing the comment in association with the image displayed on the work screen;
means for automatically generating a report using the stored comments and the associated images;
Equipped with
when the user selects an area in the image displayed on the work screen, the display means displays the area on the image , and displays the comment input tool for inputting a comment regarding the area, and further displays an auxiliary image on the work screen indicating which part of the object the area corresponds to;
the saving means saves the image displayed on the work screen, the position coordinates of the area, and the comment input using the comment input tool in association with one another;
the means for automatically creating the report creates the report in which the image, the area, and the comment are arranged based on the image, the position coordinates, and the comment that are stored in association with each other;
A system configured as follows. means for storing a report form;
the means for automatically creating the report creates the report in which the image, the area, and the comment are arranged according to a layout defined in the report form, based on the stored image, the position coordinates of the area, and the comment.
3. The system according to claim 1 or 2, configured so as to

Images