JP7464263B2 - DETECTION APPARATUS, DETECTION METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a detection device, a detection method, and a program.

Patent documents 1 to 4 disclose technology for analyzing images to detect power equipment, bird nests, etc.

JP 2019-179305 A JP 2019-032685 A JP 2014-159981 A JP 2012-093334 A

The efficiency of fault detection can be improved by photographing power equipment with a camera mounted on a moving object (vehicle, aircraft, etc.) and detecting faults through computer-based image analysis.

However, when taking a picture without human visual confirmation, there is a possibility that an unintended subject may appear in the captured image. If a computer performs image analysis on a captured image that contains such an unintended subject, there is a possibility that an erroneous judgment result will be output.

For example, in a process aimed at detecting bird nests on power equipment, if a tree is captured in the captured image, a problem may occur in which a "bird nest detected" alert is output in response to the detection of a bird nest on the tree.

The objective of the present invention is to improve the accuracy of the process of detecting defects in power equipment based on captured images.

According to the present invention,
an electric power equipment detection means for detecting electric power equipment from within a captured image and generating electric power equipment area information indicating an area within the captured image where the electric power equipment exists;
a fault detection means for detecting a fault in the power equipment by analyzing a portion of the captured image determined based on the power equipment area information;
an output means for outputting information indicating an area in the captured image in which a malfunction of the power equipment is detected;
A detection device is provided having the following:

Further, according to the present invention,
The computer
Detecting electric power equipment from within the captured image, and generating electric power equipment area information indicating an area within the captured image where the electric power equipment exists;
A portion of the captured image determined based on the electric power equipment area information is analyzed to detect a defect in the electric power equipment.
A detection method is provided that outputs information indicative of an area within the captured image in which a fault in the power equipment is detected.

Further, according to the present invention,
Computer,
an electric power equipment detection means for detecting electric power equipment from within a captured image and generating electric power equipment area information indicating an area within the captured image in which the electric power equipment exists;
a fault detection means for detecting a fault in the power equipment by analyzing a portion of the captured image determined based on the power equipment area information;
an output means for outputting information indicating an area in the captured image in which a malfunction of the power equipment is detected;
A program is provided to function as a

The present invention improves the accuracy of the process for detecting defects in power equipment based on captured images.

FIG. 2 is a diagram illustrating an example of a hardware configuration of a detection device according to the present embodiment. FIG. 2 is an example of a functional block diagram of a detection device according to the present embodiment. 5A and 5B are diagrams for explaining the processing of the detection device of the present embodiment. FIG. 2 is a diagram illustrating an example of information processed by the detection device of the present embodiment. 3A and 3B are diagrams illustrating an example of an image output by the detection device according to the present embodiment. 3A and 3B are diagrams illustrating an example of an image output by the detection device according to the present embodiment. 1A to 1C are diagrams for explaining an example of generation of an estimation model processed by the detection device of the present embodiment. 4 is a flowchart showing an example of a process flow of the detection device of the present embodiment. 5A and 5B are diagrams for explaining the processing of the detection device of the present embodiment.

First Embodiment
"overview"
The detection device of this embodiment performs a process to detect power equipment from within a captured image, and then analyzes a portion of the captured image determined based on the area within the captured image in which the power equipment was detected, to detect malfunctions in the power equipment.

With this type of detection device, only the portion of the captured image where the power equipment is present and/or its surrounding area can be analyzed for malfunction detection, and other portions can be excluded from the analysis. When configured in this manner, portions containing unintended objects that may degrade the accuracy of malfunction detection can be excluded from the analysis. As a result, it is possible to suppress the inconvenience of analyzing the portion containing the object and outputting an erroneous judgment result.

"Hardware Configuration"
Next, an example of the hardware configuration of the detection device will be described. The functional units of the detection device of this embodiment are realized by any combination of hardware and software, centering on the CPU (Central Processing Unit) of any computer, memory, programs loaded into the memory, a storage unit such as a hard disk that stores the programs (programs that are stored before the device is shipped, as well as programs downloaded from storage media such as CDs (Compact Discs) or servers on the Internet), and a network connection interface. Those skilled in the art will understand that there are various variations in the realization method and device.

Figure 1 is a block diagram illustrating the hardware configuration of the detection device of this embodiment. As shown in Figure 1, the detection device has a processor 1A, a memory 2A, an input/output I/F (interface) 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. Note that the detection device does not have to have the peripheral circuit 4A. The detection device may be configured as one device that is physically and/or logically integrated, or may be configured as multiple devices that are physically and/or logically separated. When the detection device is configured as multiple devices that are physically and/or logically separated, each of the multiple devices can have the above hardware configuration.

The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input/output interface 3A to send and receive data to each other. The processor 1A is, for example, a processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is, for example, a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input/output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, etc., and an interface for outputting information to an output device, an external device, an external server, etc. Examples of the input device are a keyboard, a mouse, a microphone, a touch panel, a physical button, a camera, etc. Examples of the output device are a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

"Function Configuration"
Next, a functional configuration of the detection device will be described. An example of a functional block diagram of the detection device 10 is shown in Fig. 2. As shown in the figure, the detection device 10 has a power equipment detection unit 11, a fault detection unit 12, and an output unit 13.

The power equipment detection unit 11 detects power equipment from within the captured image and generates power equipment area information that indicates the area within the captured image in which the power equipment exists.

In this embodiment, an estimation model for electric power equipment is prepared in advance, which, when an image is input, outputs electric power equipment area information indicating the area within the image in which the electric power equipment exists. The electric power equipment detection unit 11 detects electric power equipment and generates electric power equipment area information based on the estimation model for electric power equipment.

The estimation model for power equipment may be an estimator generated by machine learning based on training data that associates images with labels (information that identifies the power equipment, such as names). Alternatively, the estimation model for power equipment may be a model that performs template matching to match an input image with pre-prepared "template data that indicates the external features of each piece of power equipment (such as shape and color features)."

In either case, as shown in FIG. 3, multiple partial areas P are selected in the captured image, and the presence or absence of power equipment is determined by performing image analysis on each partial area P. Then, power equipment area information indicating the partial areas P in which it has been determined that power equipment exists is output. The power equipment area information is represented, for example, by coordinates in a two-dimensional coordinate system with an arbitrary position in the captured image as the origin and arbitrary directions as the x-axis and y-axis. The shape and size of the partial area P are design matters. In addition, multiple partial areas P selected in the captured image may overlap each other. By overlapping multiple partial areas P with each other, the accuracy of detecting power equipment is improved.

Figure 4 shows a schematic example of power equipment area information generated by the power equipment detection unit 11. As shown in the figure, power equipment area information is generated for each captured image (for each image file name). The power equipment area information for each captured image can indicate multiple areas (multiple partial areas P) within the captured image. The multiple areas may be separated from each other, may overlap with each other, or may be in contact with each other.

The fault detection unit 12 detects faults in the power equipment by analyzing a portion of the captured image determined based on the power equipment area information. Faults in the power equipment include the presence of foreign objects on the power equipment, damage, deterioration, and malfunctions. Foreign objects include animal nests (bird nests, etc.), balloons, garbage bags, etc. The fault detection unit 12 detects faults in these power equipment based on the external characteristics that appear in the captured image.

The fault detection unit 12 may perform image analysis on the area in the captured image indicated by the power equipment area information. Additionally, the fault detection unit 12 may perform image analysis on the area in the captured image indicated by the power equipment area information and the surrounding area.

In this embodiment, a defect estimation model is prepared in advance, which, when an image is input, outputs defect area information indicating an area in the image where a defect exists. The defect detection unit 12 realizes defect detection and generation of defect area information based on the defect estimation model.

The defect estimation model may be an estimator generated by machine learning based on training data that associates images with labels (information that identifies the defect (such as the name of the defect)). Alternatively, the estimation model may be a model that performs template matching, which matches the input image with pre-prepared "template data that indicates the external features of each defect (such as shape and color features)."

The defect detection unit 12 inputs an image of a portion of the captured image that is to be subjected to image analysis into a defect estimation model, and generates defect area information that indicates a partial area in the captured image where a defect in the power equipment exists. The defect area information is expressed, for example, in the coordinates of a two-dimensional coordinate system with an arbitrary position in the captured image as the origin and arbitrary directions as the x-axis and y-axis.

The output unit 13 outputs information indicating the area in the captured image where a malfunction in the power equipment was detected. The output is realized via an output device such as a display, a projection device, a mailer, or a printer.

For example, the output unit 13 may generate and output an image in which information indicating the area in which the malfunction of the power equipment has been detected is displayed on the captured image, as shown in FIG. 5. The information indicating the area in which the malfunction of the power equipment has been detected may be a frame W1 as shown in FIG. 5, or may be something else.

In addition, the output unit 13 may generate and output an image in which "information indicating the area indicated by the power equipment area information" and "information indicating the area in which a power equipment malfunction has been detected" are displayed on the captured image in a manner that allows them to be distinguished from each other, as shown in FIG. 6. The information indicating the area in which a power equipment malfunction has been detected may be frame W1 as shown in FIG. 6, or may be something else. Furthermore, the information indicating the area in which a power equipment malfunction has been detected may be frame W2 as shown in FIG. 6, or may be something else. Frames W1 and W2 are distinguishable from each other by making the colors, line thicknesses, line types (solid lines, dotted lines, etc.), color shadings, etc. different from each other.

When the output unit 13 receives a user input specifying either frame W1 or W2 on the displayed image as shown in Figures 5 and 6, the output unit 13 may enlarge and display the image within that frame.

When the output unit 13 displays an image in which a malfunction of the power equipment exists (such as an image in which the frame W1 is displayed), the output unit 13 may perform a warning notification process to notify the operator of the existence of the malfunction. As the warning notification process, the output unit 13 may output an alarm sound, may prominently display warning information on the screen (such as text such as "malfunction exists"), or may do something else.

The output unit 13 may also vary the configuration of the frame W1 (color, line thickness, line type, color shading, etc.) depending on the degree of certainty of the "defect detection" by the defect detection unit 12.

Here, we explain an example of a process for generating an estimation model for power equipment and an estimation model for malfunctions using machine learning.

First, when a reference image for generating training data is collected, an operator performs a labeling operation. In the labeling operation, the operator displays the reference image on a display or the like, then specifies an area in the reference image where power equipment or a defect exists, and inputs information that identifies the power equipment or defect (label: power equipment name, name of defect, etc.). At this time, the operator may further input information indicating whether it is power equipment or a defect that exists in the specified area. Through this labeling operation by the operator, training data such as that shown in FIG. 7 is generated and stored in a storage device.

At any time thereafter, the computer generates an estimation model for power equipment and an estimation model for malfunction by machine learning based on the training data as shown in FIG. 7. When generating the estimation model for power equipment, machine learning is performed using training data related to power equipment (e.g., training data whose type is "power equipment"). When generating the estimation model for malfunction, machine learning is performed using training data related to malfunctions (e.g., training data whose type is "malfunction").

Next, an example of the process flow for detecting a defect by the detection device 10 of this embodiment will be described using the flowchart in FIG.

This processing may be performed in real time while the captured image is being generated. Alternatively, this processing may be performed in batch processing at any time after the captured image is generated.

The captured image may also be generated by a camera mounted on a moving object (such as a vehicle or an aircraft). Alternatively, the captured image may be generated by a camera held by an operator.

When it is time to acquire the captured image (Yes in S10), the power equipment detection unit 11 acquires the captured image (S11) and executes a power equipment detection process (S12). In the power equipment detection process, the power equipment detection unit 11 detects power equipment from within the captured image and generates power equipment area information that indicates the area within the captured image in which the power equipment is present. The details of the power equipment detection process are as described above, so a description thereof will be omitted here.

If no power equipment is detected in the captured image (No in S13), and there is no input to end the process (No in S16), the process returns to S10 and the same process is repeated.

On the other hand, if power equipment is detected from the captured image (Yes in S13), the malfunction detection unit 12 analyzes a portion of the captured image determined based on the power equipment area information generated in S12 to detect a malfunction in the power equipment (S14). The details of the malfunction detection process are as described above, so a description thereof will be omitted here.

Then, the output unit 13 outputs information indicating the area in the captured image where the malfunction of the power equipment was detected (S15). For example, the output unit 13 may output an image in which information indicating the area where the malfunction of the power equipment was detected (frame W1) is displayed on the captured image as shown in FIG. 5. Alternatively, the output unit 13 may output an image in which "information indicating the area indicated by the power equipment area information (frame W2)" and "information indicating the area where the malfunction of the power equipment was detected (frame W1)" are displayed on the captured image in a manner that allows them to be distinguished from each other as shown in FIG. 6.

"Action and effect"
The detection device 10 of this embodiment performs a process of detecting power equipment from within a captured image, and then analyzes a portion of the captured image determined based on the area within the captured image in which the power equipment was detected, to detect malfunctions in the power equipment.

With this detection device 10, only the portion of the captured image where the power equipment is present and/or its surrounding area can be analyzed for malfunction detection, and other portions can be excluded from the analysis. When configured in this manner, portions containing unintended objects that may degrade the accuracy of malfunction detection can be excluded from the analysis. As a result, it is possible to suppress the inconvenience of analyzing the portion containing the object and outputting an erroneous judgment result.

Furthermore, according to the detection device 10 of this embodiment, as shown in FIG. 6, it is possible to output an image in which "information indicating the area indicated by the power equipment area information (frame W2)" and "information indicating the area in which a malfunction of the power equipment has been detected (frame W1)" are displayed on the captured image in a manner that allows them to be distinguished from each other. By viewing such an image, the operator can easily grasp not only the presence and location of a malfunction, but also which part of the captured image has been detected as the part in which the power equipment is present and has become the subject of analysis for malfunction detection. As a result, the operator can grasp not only the presence or absence and location of a malfunction, but also whether all of the power equipment has been detected in the captured image and analyzed for malfunction detection.

Second Embodiment
As described in the first embodiment, the detection device 10 of this embodiment selects multiple partial areas P in the captured image as shown in Fig. 3 and performs image analysis on each partial area P to determine the presence or absence of power equipment. The detection device 10 then generates power equipment area information indicating the partial area P in which it has been determined that power equipment exists. The multiple partial areas P selected in the captured image overlap with each other, and the greater the degree of overlap, the higher the detection accuracy of the power equipment.

Under these circumstances, if a portion of a captured image determined based on power equipment area information is analyzed to detect defects in the power equipment without any special ingenuity, problems may arise in the form of increased processing burden on the computer and increased processing time.

For example, consider a case where a portion of an image in a captured image corresponding to each of a plurality of areas (a plurality of partial areas P in which it is determined that power equipment exists) indicated by the power equipment area information is cut out, and each of the cut-out images is used as an analysis target for defect detection. As described above, the plurality of partial areas P selected in the captured image overlap each other, so the plurality of areas indicated by the power equipment area information may also overlap each other. In this case, the contents of the plurality of images cut out corresponding to each of the plurality of areas may overlap. If defect detection analysis is performed on each of the plurality of images with such overlapping contents, the same location in the captured image will be overlapped and analyzed, which increases the processing burden on the computer and the time required for processing. For example, if the plurality of areas indicated by the power equipment area information overlap each other M times, the inconvenience of overlapping and analyzing any position in the captured image M times may occur.

The detection device 10 of this embodiment is equipped with a means to solve this problem.

The defect detection unit 12 performs defect detection based on the defect estimation model described in the first embodiment.

The process of the fault detection unit 12 will be described in detail with reference to FIG. 9. First, the fault detection unit 12 selects multiple partial regions P' in the captured image, similar to the case of estimation based on the estimation model for power equipment described in the first embodiment (see FIG. 3 in which multiple partial regions P are selected). The shape and size of the partial regions P' are design matters. The multiple selected partial regions P' may overlap each other. By overlapping the partial regions P' with each other, the accuracy of fault detection is improved. Note that the shapes and sizes of the partial regions P and P' may be the same or different from each other.

Next, the fault detection unit 12 determines whether each of the selected partial areas P' overlaps with an area in the captured image indicated by the power equipment area information (the area surrounded by frame F in FIG. 9).

Then, the fault detection unit 12 analyzes the image of the partial area P' that overlaps with an area in the captured image indicated by the power equipment area information among the multiple selected partial areas P' to detect faults in the power equipment. Note that the fault detection unit 12 does not perform fault detection analysis on the image of the partial area P' that does not overlap with an area in the captured image indicated by the power equipment area information among the multiple selected partial areas P'.

The other configurations of the detection device 10 of this embodiment are the same as those of the first embodiment.

The detection device 10 of this embodiment achieves the same effect as the detection device 10 of the first embodiment. Furthermore, the detection device 10 of this embodiment performs image analysis only on the image of the partial area P', among the multiple partial areas P' selected in the captured image, that overlaps with the area in the captured image indicated by the power equipment area information. With this detection device 10, when multiple areas indicated by the power equipment area information overlap each other M times, the inconvenience of having to overlap and analyze any position in the captured image M times can be avoided. As a result, the processing burden on the computer and the time required for processing can be reduced.

<Modification>
Here, a modified example applicable to all the embodiments will be described. The electric power equipment detection unit 11 may detect electric power equipment from the captured image by using a method for detecting a pixel area in which a detection target exists, which is called semantic segmentation or instance segmentation. Even when the processing of the electric power equipment detection unit 11 is modified in this way, the same effect as that of the first and second embodiments described above is achieved.

In this specification, "acquisition" may include "the device itself going to retrieve data stored in another device or storage medium (active acquisition)" based on user input or program instructions, such as receiving data by making a request or inquiry to another device, or accessing and reading out another device or storage medium. Also, "acquisition" may include "inputting data output from another device to the device itself (passive acquisition)" based on user input or program instructions, such as receiving data that is distributed (or transmitted, push notification, etc.). Also, "acquisition" may include selecting and acquiring data or information received, and "generating new data by editing data (converting to text, rearranging data, extracting some data, changing the file format, etc.), and acquiring the new data."

A part or all of the above-described embodiments can be described as follows, but is not limited to the following.
1. A power equipment detection means for detecting power equipment from within a photographed image and generating power equipment area information indicating an area within the photographed image in which the power equipment exists;
a fault detection means for detecting a fault in the power equipment by analyzing a portion of the captured image determined based on the power equipment area information;
an output means for outputting information indicating an area in the captured image in which a malfunction of the power equipment is detected;
A detection device having the following:
2. The detection device according to 1, wherein the malfunction detection means detects an animal nest present on the power equipment.
3. The defect detection means
Selecting a plurality of partial areas in the captured image;
determining whether each of the plurality of partial regions overlaps with a region in the captured image indicated by the power equipment region information;
An image of the partial area overlapping with an area in the captured image indicated by the electric power equipment area information is analyzed to detect a malfunction of the electric power equipment;
A detection device as described in 1 or 2, which does not analyze the image of the partial area that does not overlap with the area in the captured image indicated by the power equipment area information.
4. The power equipment detection means
generating electric power equipment area information indicating a plurality of areas in the captured image in which the electric power equipment exists;
4. The detection device according to claim 3, wherein the multiple areas indicated by the power equipment area information may overlap with each other.
5. The detection device according to any one of 1 to 4, wherein the output means generates and outputs an image in which an area indicated by the electric power equipment area information and an area in which a malfunction of the electric power equipment has been detected are displayed on the captured image in a manner that allows them to be distinguished from each other.
6. The computer:
Detecting electric power equipment from within the captured image, and generating electric power equipment area information indicating an area within the captured image where the electric power equipment exists;
A portion of the captured image determined based on the electric power equipment area information is analyzed to detect a defect in the electric power equipment.
A detection method that outputs information indicating an area in the captured image in which a malfunction of the power equipment is detected.
7. Computers,
an electric power equipment detection means for detecting electric power equipment from within a captured image and generating electric power equipment area information indicating an area within the captured image in which the electric power equipment exists;
a fault detection means for detecting a fault in the power equipment by analyzing a portion of the captured image determined based on the power equipment area information;
an output means for outputting information indicating an area in the captured image in which a malfunction of the power equipment is detected;
A program that functions as a

The present invention has been described above with reference to the embodiments (and examples), but the present invention is not limited to the above-mentioned embodiments (and examples). Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

1A Processor 2A Memory 3A Input/Output I/F
4A Peripheral circuit 5A Bus 10 Detection device 11 Power equipment detection unit 12 Malfunction detection unit 13 Output unit

Claims (7)

an electric power equipment detection means for detecting electric power equipment from within a captured image and generating electric power equipment area information indicating an area within the captured image in which the electric power equipment exists;
a fault detection means for detecting a fault in the power equipment by analyzing a portion of the captured image determined based on the power equipment area information;
an output means for outputting information indicating an area in the captured image in which a malfunction of the power equipment is detected;
having
The defect detection means includes:
A detection device that analyzes an image of a partial area selected from a plurality of partial areas in the captured image that overlaps with an area in the captured image indicated by the power equipment area information, and detects malfunctions in the power equipment . The detection device according to claim 1, wherein the malfunction detection means detects an animal nest present on the power equipment. The defect detection means includes:
Selecting a plurality of partial areas in the captured image;
determining whether each of the plurality of partial regions overlaps with a region in the captured image indicated by the power equipment region information;
An image of the partial area overlapping with an area in the captured image indicated by the electric power equipment area information is analyzed to detect a malfunction of the electric power equipment;
The detection device according to claim 1 or 2, wherein the image of the partial area that does not overlap with the area in the captured image indicated by the power equipment area information is not analyzed. The power equipment detection means
generating electric power equipment area information indicating a plurality of areas in the captured image in which the electric power equipment exists;
The detection device according to claim 3 , wherein the plurality of areas indicated by the power equipment area information can overlap with each other. The detection device according to any one of claims 1 to 4, wherein the output means generates and outputs an image in which the area indicated by the power equipment area information and the area in which the power equipment malfunction has been detected are displayed on the captured image in a manner that allows them to be distinguished from each other. The computer
Detecting electric power equipment from within the captured image, and generating electric power equipment area information indicating an area within the captured image where the electric power equipment exists;
A portion of the captured image determined based on the electric power equipment area information is analyzed to detect a defect in the electric power equipment.
outputting information indicating an area in the captured image in which a malfunction of the power equipment is detected ;
In the process of detecting the defect,
A detection method for detecting malfunctions in the power equipment by analyzing an image of a partial area selected from a plurality of partial areas in the captured image that overlaps with an area in the captured image indicated by the power equipment area information . Computer,
an electric power equipment detection means for detecting electric power equipment from within a captured image and generating electric power equipment area information indicating an area within the captured image in which the electric power equipment exists;
a fault detection means for detecting a fault in the power equipment by analyzing a portion of the captured image determined based on the power equipment area information;
an output means for outputting information indicating an area in the captured image in which a malfunction of the power equipment is detected;
Function as a
The defect detection means includes:
A program that analyzes an image of a partial area selected from a plurality of partial areas in the captured image that overlaps with an area in the captured image indicated by the power equipment area information, and detects malfunctions in the power equipment .

Images