JP7809565B2 - Method for acquiring data for inspecting wind turbines, program for causing a computer to execute the method, and unmanned aerial vehicle - Google Patents

Description

This disclosure relates to techniques for inspecting wind turbine blades, and more particularly to techniques for acquiring data for inspecting blades.

Technology for inspecting the blades of wind turbines used in wind power generation is known. For example, Japanese Patent Application Laid-Open No. 2017-090145 (Patent Document 1) discloses "a wind turbine blade deformation measurement device that can easily and accurately determine blade stiffness without requiring special synchronization between sensor and wind speed information acquisition."

Japanese Patent Application Laid-Open No. 2017-090145

Wind turbine blades are several tens of meters long, so when capturing images to inspect their appearance, it is not possible to capture all the necessary images in one shot, and multiple images must be taken. On the other hand, because wind turbines have three blades, even if an image of part of a blade is captured, it may not be clear which blade the image corresponds to. Therefore, there is a need for technology that makes it easy to determine which blade the image refers to.

This disclosure has been made in light of the above-mentioned background, and one of its purposes is to disclose technology that can identify which blade is included in multiple images.

(1) According to one embodiment, a method for acquiring data for inspecting a wind turbine is provided. This method includes the steps of: using one or more cameras mounted on an unmanned aerial vehicle to simultaneously photograph a pre-identified portion of the hub of the wind turbine and a portion of one of a plurality of blades attached to the hub multiple times; repeating the simultaneous photographing step for the remaining blades; and identifying the photographed blade based on the presence or absence of the pre-identified portion or the orientation of the portion in an image based on the image data acquired by photographing, and the positional relationship of the plurality of blades to the pre-identified portion.

(2) In one aspect, the step of simultaneously photographing multiple times includes the steps of simultaneously photographing a pre-specified portion of the hub with a first camera of the one or more cameras and photographing a portion of one blade with a second camera of the one or more cameras, and, in response to the simultaneous photographing of the pre-specified portion and portion, associating the image data acquired by the first camera and the image data acquired by the second camera with the time at which the photographs were taken.

(3) In one aspect, the step of simultaneously capturing images multiple times includes acquiring still images. The step of associating the time includes associating the time with the image data acquired by the first camera and the image data acquired by the second camera.

(4) In one aspect, the step of simultaneously capturing images multiple times includes acquiring video images. The step of associating the time includes associating the time with frames included in the video image data acquired by the first camera and frames included in the video image data acquired by the second camera.

(5) In one aspect, the step of simultaneously photographing multiple times includes a step of directing the photographing direction of the first camera toward a predetermined part of the body so that the predetermined part does not fall outside the photographing range in accordance with the movement of the unmanned aerial vehicle.

(6) In one aspect, cameras are provided on the top and bottom of the housing of the unmanned aerial vehicle. The simultaneous photographing step includes, when the blade to be photographed is located above the hub, using the camera provided on the top as a second camera to photograph a portion of one blade, and using the camera provided on the bottom as a first camera to photograph a pre-specified portion.

(7) In one aspect, cameras are provided at the top and bottom of the housing of the unmanned aerial vehicle. The simultaneous photographing step includes, when the blade to be photographed is located below the hub, using the camera provided at the bottom as a first camera to photograph a portion of one blade, and using the camera provided at the top as a second camera to photograph a pre-specified portion.

(8) In one aspect, the acquisition method further includes a step of identifying the blade to which the image of a portion of a blade acquired at the time an image including a predetermined portion is acquired.

(9) In one aspect, the identifying step includes a step of analyzing the multiple images and extracting pre-identified parts from each image, and a step of labeling the blades included in the image of a portion of one blade acquired at the time each image was acquired, based on blade identification information that has been pre-identified based on the positional relationship between the predetermined parts and the multiple blades.

(10) In one aspect, the step of taking multiple simultaneous photographs includes taking simultaneous photographs while the unmanned aerial vehicle is hovering while the blades are operating.

(11) In one aspect, the step of taking multiple simultaneous photographs includes taking simultaneous photographs while the unmanned aerial vehicle is moving while the blade is stationary.

(12) In some aspects, moving the unmanned aerial vehicle includes moving the unmanned aerial vehicle from the tip of the blade to be photographed toward the hub, or moving the unmanned aerial vehicle from the hub toward the tip of the blade to be photographed.

(13) According to another embodiment, a program is provided. This program causes a computer to execute any of the acquisition methods described above.

(14) According to yet another embodiment, an unmanned aerial vehicle is provided. The unmanned aerial vehicle includes a memory storing the above-described program, one or more cameras disposed on the unmanned aerial vehicle, and a control device that executes the program to control the unmanned aerial vehicle.

In one embodiment, it is possible to identify which blade appears in multiple images, making visual inspection of the blade easier.

The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings.

FIG. 1 is a diagram showing an overview of a method for inspecting the exterior of an operating wind turbine 10 using a drone 100 as an air vehicle. 1 is a diagram showing a portion of the path that the drone 100 flies when inspecting the appearance of three blades 120, 121, and 122. FIG. 1 is a functional block diagram illustrating the configuration of the drone 100. FIG. 1 is a diagram illustrating an example of the appearance of a drone 100. FIG. 10 is a diagram illustrating a situation in which the drone 100 takes multiple photographs to inspect a single blade. FIG. 1 is a side view of the wind turbine 10. 7 is a diagram showing an example of an image 700 of a part of the wind turbine 10 taken from below the wind turbine 10. FIG. A figure showing an example of data stored in a recording medium 38 possessed by the drone 100. FIG. 10 is a diagram illustrating an example of a state in which a nacelle 160 is photographed from above the wind turbine 10. FIG. 1 is a diagram illustrating an example of the hardware configuration of a computer device 1000 that functions as an image analysis device. FIG. 11 is a diagram illustrating an example of an image 1100 taken from above the wind turbine 10. 10 is a flowchart showing a part of a process performed to identify blades from a still image of the wind turbine 10. 10 is a flowchart showing a part of a process performed to identify blades from a moving image of the wind turbine 10. 10 is a flowchart showing a part of a process performed to identify a blade from a moving image and a still image. 10 is a flowchart showing a part of the process performed when a single camera is used to photograph the hub 140 and the blades. 1 shows an image obtained by first camera 18 photographing hub 140. 17 shows an image 1700 showing the tip of the blade 120 identified in step S1225.

Embodiments of the present invention will now be described with reference to the drawings. In the following description, identical components are designated by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions of them will not be repeated.

[Outline of visual inspection using drones]
A method for inspecting a wind turbine 10 using a drone 100 according to an embodiment will be described with reference to Figures 1 and 2. Figure 1 is a diagram showing an overview of a method for inspecting the exterior of an operating wind turbine 10 using a drone 100 as an air vehicle.

The wind turbine 10 is installed on a foundation 180, which is located in a location with a relatively high wind volume, for example, offshore or onshore. The wind turbine 10 includes blades 120, 121, 122, a hub 140, and a tower 130. The hub 140 is provided with a hatch 150.

The drone 100 is used to inspect the exterior of the wind turbine 10. The drone 100 is equipped with a first camera 18 that can capture images above or in front of the drone 100, a second camera 20 that can capture images below or in front of the drone 100, and a LiDAR (Laser Imaging Detection and Ranging) sensor 24.

Drone 100 approaches near wind turbine 10 without coming into contact with blades 120, 121, and 122, moves from bottom to top, top to bottom, left to right, or right to left, and stops at the shooting location, taking synchronized images using first camera 18 and second camera 20. Drone 100 drives one of the two cameras to maintain a position that allows it to photograph a predetermined area (e.g., hatch 150, a marker marked as an inspection target, etc.). Drone 100 uses the other camera to photograph the exterior of each blade from the tip toward the center, or from the center toward the tip, while synchronizing the timing of the images taken by first camera 18 and second camera 20. By using the two cameras to capture images simultaneously, it is possible to correlate the images of hatch 150 and the exterior of the blades at each timing. Furthermore, because each captured image is associated with a captured image of the hatch 150, the blade's positional relationship to the hatch 150 becomes clear, making it easy to identify which blade is the one being photographed.

Once photography at a certain location is completed, the drone moves to the next location and takes a similar photograph. The image data obtained by the photography is stored in the drone 100, and image analysis is then performed by a computer device. In another aspect, the drone 100 may transmit the image data to a computer device on the ground. In this case, the computer device can begin image analysis before the drone 100 returns, allowing inspection results to be obtained earlier. The detailed configuration of the drone 100 will be described later.

Note that photographing the blades 120, 121, and 122 using the drone 100 can be achieved both when the wind turbine 10 is stopped and when it is operating. Photographing the blades may also be done while the drone 100 is hovering without moving. If photographing can be done while the wind turbine 10 is operating, image data for visual inspection can be obtained while preventing a decrease in the amount of power generated.

Figure 2 shows part of the flight path of drone 100 when inspecting the appearance of three blades 120, 121, and 122. Drone 100 takes off from the ground and approaches close to the tip of blade 120, the first inspection target (path 210, drone 100A). Drone 100 takes photographs at an initial position close to blade 120 and simultaneously photographs hatch 150. The photographs may be either still or video.

For example, the drone 100 simultaneously photographs the hatch 150 using the first camera 18 and a portion (tip) of the blade 120 using the second camera 20. A timestamp indicating the time of photographing is associated with each photographed image. This photographing time is obtained, for example, using the GPS function of the drone 100. The image data generated from photographing by the first camera 18 and the image data generated from photographing by the second camera 20 are associated by this timestamp. Furthermore, one of the two cameras is always photographing the hatch 150, and the position of the hatch 150 determines which of the three blades 120, 121, or 122 the blade photographed at that time is.

The drone 100 then moves to the next position near the blade 120 and takes a similar photograph. Each position at which the drone 100 stops for photographing is set so that the captured images partially overlap, taking into account the angle of view of the camera that continues to photograph the blade 120 (second camera 20 in the above example). Meanwhile, the first camera 18 also captures the area including the hatch 150 in the next photograph taken after the drone 100 has moved. In this way, by having one of the two cameras always photograph the hatch 150, the blade can be identified based on the orientation of the hatch 150, making it easier to reconstruct an overall image of the blade 120's appearance.

When the drone 100 moves along the blade 120 to the vicinity of the hub 140 (route 220), it begins photographing the second inspection target, blade 121 (drone 100B). The drone 100 repeatedly photographs the blade 121 as it moves from the base (near the hub 140) toward the tip. After finishing photographing the blade 121, the drone 100 moves to the tip of the blade 122, the third inspection target (route 230, drone 100C). The drone 100 repeatedly moves and stops from the tip of the blade 122 toward the hub 140, repeatedly photographing the appearance of the blade 122. After finishing photographing the blade 122, the drone 100 returns to the ground (route 240).

When photographing the blade 120 shown in FIG. 2, the drone 100 may photograph the blade 120 while moving from top to bottom (i.e., from the hub 140 to the tip of the blade 120). In other situations, the drone 100 may photograph the grade while moving from bottom to top. If the time of day when the drone 100 is photographed may be backlit by sunlight, the drone 100 can capture a good image of the blade surface by photographing while moving from top to bottom or from an angle.

On the ground, the image data acquired by the drone 100 is read and input into a computer that performs image analysis. The computer arranges the image data in chronological order based on the timestamps, and displays images of adjacent areas in sequence. Furthermore, because the blade is identified based on the orientation of the hatch 150 in the image, the computer can indicate which blade each image represents. Therefore, an inspector viewing the computer's monitor can visually inspect each of the chronologically arranged images to perform a visual inspection of each blade. Visual inspection may also be performed using AI (artificial intelligence).

In another aspect, each image data may be transmitted wirelessly to a computer device on land while the drone 100 is flying. In this case, a visual inspection of any blade whose entire image has been captured can be performed in advance while the drone 100 is flying.

[Drone configuration]
The configuration of the drone 100 will be described with reference to Figures 3 and 4. Figure 3 is a functional block diagram illustrating the configuration of the drone 100. Figure 4 is a diagram illustrating the external appearance of the drone 100.

As shown in FIG. 3 , the drone 100 includes a main body 11, four propeller motors 13, a motor drive circuit 15, an engine 16, a generator 17, a first camera 18, a gimbal 19, a second camera 20, a gimbal 21, and a LiDAR sensor 24. The main body 11 includes a controller 30, a memory device 31, an inertial measurement unit 32, a transceiver 33, a GPS (Global Positioning System) receiver 34, a power supply circuit 35, and a storage battery 36. The storage device 31 has a reader/writer 37 and a recording medium 38. If the drone 100 is electrically powered and does not include the engine 16 and generator 17, the storage battery 36 drives the propeller motors 13.

The generator 17 is driven by the engine 16 to generate the electricity required to operate the drone 100. This enables the drone 100 to fly for long periods of time and in low temperatures.

The first camera 18 is attached to the top surface of the main body 11 so that it can capture images in front of or above the drone 100. The first camera 18 can change its capture direction between horizontal and upward using an actuator driven by an external signal. The actuator may also be rotatable around a vertical axis. The actuator may also be absent.

The second camera 20 is attached to the underside of the main body 11 so that it can capture images in front of or below the drone 100. In one aspect, the second camera 20 is positioned so that it captures images in the direction of travel of the drone 100. In another aspect, the second camera 20 may be attached to the main body 11 so that it can rotate around the vertical axis.

The first camera 18 may include a high-resolution camera 18A for capturing inspection images and a low-resolution camera 18B for capturing video images for FPV (First Person View). The second camera 20 may include a high-resolution camera 20A for capturing inspection images and a low-resolution camera 20B for capturing FPV images. Video images captured by the low-resolution cameras 18B and 20B are transmitted to a wireless communication device (not shown) at the base facility. This allows inspectors at the base facility to view the images captured by cameras 18B and 20B in real time. The shooting directions of the first camera 18 and the second camera 20 are adjusted by the actuators that make up the gimbals 19 and 21, respectively. The actuators that make up the gimbals 19 and 21 can also be adjusted by signals from the wireless communication device at the base facility.

Gimbal 19, 21 are electric actuators that can adjust the orientation of first camera 18 and second camera 20 in three dimensions. Based on the detection values of the acceleration sensor and angular velocity sensor, gimbals 19, 21 can keep the imaging direction of first camera 18 and second camera 20 constant, even if the attitude of drone 100 changes.

The LiDAR sensor 24 detects the distance and angle to an object based on the light reflected from the object by scanning with a pulsed laser light. Other distance sensors may be used instead of the LiDAR sensor 24.

The controller 30 controls the operation of the motor drive circuit 15, first camera 18, second camera 20, LiDAR sensor 24, etc. The controller 30 may be configured with a microcomputer including a CPU (Central Processing Unit), RAM (Random Access Memory), and non-volatile memory, or may be configured with an FPGA (Field Programmable Gate Array), or may be configured with a dedicated circuit such as an ASIC (Application Specific Integrated Circuit). The controller 30 may also be configured with a combination of at least two of these.

The storage device 31 includes, for example, a removable non-volatile recording medium 38 such as an SD memory card, and a reader/writer 37 for writing data to and reading data from the recording medium 38. In accordance with commands from the controller 30, the reader/writer 37 stores still images for inspection taken by the first camera 18, the control details of the controller 30, flight information, and the like, on the recording medium 38.

The inertial measurement unit 32 is a sensor unit that integrates an acceleration sensor, angular velocity sensor (gyro sensor), geomagnetic sensor, barometric pressure sensor, temperature sensor, and other sensors into a single package. The controller 30 controls the autonomous flight and attitude of the drone 100 based on the detection values of the various sensors in the inertial measurement unit 32.

Transceiver 33 transmits and receives signals, data, and other information to and from transceivers installed on the wind turbine being inspected and to transceivers on the terminal device at the onshore base facility. For example, transceiver 33 receives information about the operating status of the wind turbine and commands from the terminal device at the base facility. Transceiver 33 also transmits monitoring video images captured by low-resolution cameras 18B and 20B to the base facility in accordance with commands from controller 30.

The GPS receiver 34 receives signals transmitted from each of the multiple GPS satellites. The controller 30 detects the current position of the drone 100 based on the signals received by the GPS receiver 34.

The power supply circuit 35 generates a power supply voltage to drive each part of the drone 100 based on the power generated by the generator 17.

The storage battery 36 is used as an auxiliary power source for the drone 100. For example, the storage battery 36 supplies power to drive the controller 30, memory device 31, transceiver 33, etc. when the engine 16 and generator 17 are stopped.

Figure 4(A) is a view of the drone 100 from the front. Figure 4(B) is a view of the drone 100 from the side.

As shown in Figure 4 (A), the drone 100 has four sets of propeller motors 13 and propellers 14. Each propeller motor 13 is connected to the main body 11 via an arm 12. The legs 22 are connected to the lower part of the main body 11.

While Figure 4 illustrates a drone 100 equipped with four sets of propeller motors 13 and propellers 14, the flight mechanism of the drone 100 is not limited to the illustrated configuration. In other aspects, by controlling the rotational speed of each propeller motor 13, the drone 100 can ascend and descend vertically, ascend and descend in any diagonal direction, hover in mid-air, move forward, backward, move right, move left, rotate right, rotate left, and so on. In other words, the propeller motors 13 and propellers 14 can form a propulsion mechanism 23 that propels the drone 100 and stops it in mid-air.

[Filming Overview]
A method of photographing by drone 100 according to this embodiment will be described with reference to Figures 5 and 6. Figure 5 is a diagram illustrating a situation in which drone 100 photographs multiple times to inspect a single blade.

Figure 5 shows a wind turbine 10 viewed from the front. An example is shown in which the drone 100 captures images of a single blade from ten locations. The number of images taken is calculated in advance by the user of the drone 100, for example, based on the imaging range and resolution of the first camera 18 and second camera 20 mounted on the drone 100. The number of images taken is calculated as the number of times the edges of the captured images of the surface of a single blade are adjacent, so that an image of the appearance of a single blade can be reconstructed from each image.

After taking off from the ground, the drone 100 approaches the tip of the blade 120, which is the first inspection target, and at position 100-1, photographs a portion of the blade 120 and the hatch 150. The drone 100 then moves to position 100-2. At position 100-2, the drone 100 simultaneously photographs a portion of the blade 120 and the hatch 150, similar to the photographing at position 100-1. The drone 100 then moves to the next position as indicated by the arrow, and again simultaneously photographs a portion of the blade 120 and the hatch 150. The drone 100 then moves again, and at the final position 100-10, performs simultaneous photographing. After finishing photographing the blade 120, the drone 100 moves to the tip of the next blade and begins similar simultaneous photographing. The drone 100 can also take simultaneous photographs while moving from the blade's attachment point toward the tip, without moving to the tip of the next blade. After completing multiple simultaneous photographs of each of the three blades, the drone 100 returns to the ground. Photographing can be performed using both the first camera 18 and the second camera 20, or just one of them.

Depending on the position of the drone 100, the hatch 150 may not be visible to the drone 100. Therefore, the image data obtained by photographing each blade may include image data of the hatch 150 and image data of part of the blade, or may not include image data of the hatch 150. Because the positional relationship between the hatch 150 and each blade is known in advance, it is possible to identify which blade a part of a blade photographed simultaneously belongs to. Therefore, image data obtained by photographing a single blade multiple times can be reconstructed by image processing on a computer device.

Figure 6 is a side view of the wind turbine 10. At position 610, the drone 100 measures the distance to the front of the tower 130. In this case, the drone 100 determines position 610 so that the distance is greater than the distance between the blades 120 and the tower 130. This prevents collisions between the drone 100 and the blades 120 during imaging at position 610.

The drone 100 measures the distance to the hub 140 from a position in front of the hub 140 (position 620). Near the hub 140, the drone 100 and the blades do not interfere with each other, so the drone 100 can approach closer to the front of the hub 140. As an example, the distance between the drone 100 and the hub 140 is shorter than the distance between the drone 100 and the tower 130. The camera used to capture the front of the hub 140 may be either the first camera 18 or the second camera 20.

The drone 100 may also photograph the rear of the wind turbine 10. For example, the drone 100 measures the distance to the nacelle 160, approaches the nacelle 160, and photographs the exterior of the nacelle 160 (position 630). The camera used to photograph the nacelle 160 may be either the first camera 18 or the second camera 20.

Furthermore, the drone 100 may also photograph the rear of the tower 130. In this case, the drone 100 measures the distance to the tower 130, approaches the tower 130, and photographs the exterior of the tower 130 (position 640). The camera used to photograph the tower 130 may be either the first camera 18 or the second camera 20.

[Example of a captured image]
An example of a captured image will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of an image 700 captured of a part of the wind turbine 10 from below the wind turbine 10. The image 700 includes the blades 120, 121, 122, the tower 130, the hub 140, the hatch 150, and the nacelle 160.

Since image 700 includes hatch 150, by considering the position of the blade relative to hatch 150, it is easy to identify which blade is included in the images captured at the same time.

[Data Structure]
The structure of data acquired by the drone 100 will now be described with reference to Fig. 8. Fig. 8 is a diagram illustrating one example of data stored in the recording medium 38 of the drone 100. The drone 100 associates two sets of image data acquired by simultaneous shooting using the first camera 18 and the second camera 20 with time information and writes the data to the recording medium 38.

For example, image data 810 includes an image of hatch 150 captured by first camera 18. The image data 810 was captured at 11:22:00 on March 15, 2022. Based on the orientation of hatch 150, the blade visible below hub 140 is identified as blade 121. On the other hand, image data 811 includes an image of blade 121 captured by second camera 20. The image data 811 was also generated at 11:22:00 on March 15, 2022. As an example, images are captured by drone 100 at two-second intervals, and the timestamps of each image data are matched to the nearest second.

Similarly, image data 820 and image data 821 are associated with the same time information, and therefore image data 820 and image data 821 are data obtained by photographing at the same time.

Because the timestamps of image data 810 and 811 are the same, and the timestamps of image data 820 and 821 are the same, the computer device performing the image analysis can associate image data 810 with image data 811, and image data 820 with image data 821, and display them on a monitor. This allows the inspector to easily recognize that blade 121 has been photographed, taking into account the orientation of hatch 150 shown in image data 810 or 820 and the positional relationship between each blade.

[Shooting from above]
Another imaging method will be described with reference to Fig. 9. Fig. 9 is a diagram illustrating an example of how the nacelle 160 is imaged from above the wind turbine 10.

Depending on the location and time of day where the wind turbine 10 is installed, the drone 100 may be subject to backlight when photographing the wind turbine 10 from below. Therefore, to avoid backlighting, the drone 100 may simultaneously photograph the wind turbine 10 from above. In this case, the drone 100 photographs at least the hub 140 so that the hatch 150 is included in the photographed image. The drone 100 then photographs the blades at the same time as photographing the hub 140.

In the example shown in Figure 9, the drone 100 simultaneously photographs the blades and the hub 140 while moving from above to below the wind turbine 10. The drone 100 can simultaneously photograph both the upward direction (Figure 5) and the downward direction (Figure 9), ensuring that simultaneous photographs are taken in a direction that is not backlit.

The orientation of the nacelle 160 is calculated, for example, as follows.
[First calculation method]
(1) The photographing position, direction, and photographing angle by the drone 10 for calculating the orientation of the nacelle 160 are constant relative to the wind turbine 10.
(2) Images of the nacelle 160 in various directions are captured in advance, and a deep learning model is constructed using each of the images as training data. The training data is, for example, images obtained by capturing images of the nacelle 160 from the same angle at the same altitude. The deep learning model is constructed, for example, using a convolutional neural network or the like.
(3) The deep learning model is installed on a computing device (e.g., computer device 1000 in Figure 10) within the drone 10 body or flight management base.
(4) After the drone 10 takes a photograph at the photographing position for calculating the orientation of the nacelle 160, the image data obtained by the photographing is input into the calculation device.
(5) The inspector executes the nacelle orientation calculation program from the user interface screen of the flight management program of the computing device and obtains the estimated results of the nacelle orientation.

[Second calculation method]
(1) The photographing position, direction, and photographing angle by the drone 10 for calculating the orientation of the nacelle 160 are constant relative to the wind turbine 10.
(2) The rotation (yaw) plane of the nacelle 160 is constant on the captured image (two-dimensional projection surface).
(3) From an image of the nacelle 160 photographed by the drone 10 from a predetermined position, the structurally specific characteristic shape of the nacelle 160 (for example, the direction of the main body that houses the rotation shaft of the hub 140 or the direction of the bottom surface of the main body) is captured as a line segment, and the actual angle (orientation) is geometrically calculated from the angle of the line segment on the photographed image, and then the nacelle 160 is calculated. The direction of the characteristic shape can be specified by the inspector.

[Third calculation method]
(1) The drone 100 flies around the wind turbine 10 while facing the wind turbine 10, maintaining a constant distance at the same altitude as the nacelle 160.
(2) While monitoring the image transmitted from the camera of drone 100, the inspector changes drone 100 into a hovering state when drone 100 reaches a position facing the same direction as nacelle 160 or a position directly opposite.
(3) The computing device used by the inspector displays the drone's current orientation on the flight management user interface screen, and the inspector reads the orientation of the nacelle 160 from that orientation.

[Example of image analysis device]
Next, an image analysis device that analyzes the appearance of a wind turbine according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram illustrating an example of the hardware configuration of a computer device 1000 that functions as the image analysis device.

Computer device 1000 mainly comprises the following components: a CPU (Central Processing Unit) 1 that executes programs; a mouse 2 and keyboard 3 that receive instructions input by the user of computer device 1000; RAM 4 that volatilely stores data generated by program execution by CPU 1 or data input via mouse 2 or keyboard 3; a hard disk 5 that non-volatilely stores data; an optical disk drive 6; a communications interface (I/F) 7; and a monitor 8. Each component is interconnected by a data bus. A CD-ROM 9 or other optical disk is inserted into optical disk drive 6.

Processing in the computer device 1000 is realized by software executed by each piece of hardware and CPU 1. Such software may be pre-stored on the hard disk 5. Alternatively, the software may be stored on a CD-ROM 9 or other recording medium and distributed as a computer program. Alternatively, the software may be provided as a downloadable application program by an information provider connected to the Internet. Such software is read from the recording medium by an optical disk drive 6 or other reading device, or downloaded via the communication interface 7, and then temporarily stored on the hard disk 5. The software is read from the hard disk 5 by CPU 1 and stored in RAM 4 in the form of an executable program. CPU 1 executes the program.

The components constituting the computer device 1000 shown in FIG. 10 are common. Therefore, one essential part of the technical concept of this disclosure can be said to be software stored in RAM 4, hard disk 5, CD-ROM 9, or other recording media, or software that can be downloaded via a network. The recording media may include non-transitory computer-readable data recording media. Note that the operation of each piece of hardware in the computer device 1000 is well known, so a detailed description will not be repeated.

Note that recording media are not limited to CD-ROMs, FDs (Flexible Disks), and hard disks, but may also include media that permanently carry programs, such as magnetic tape, cassette tapes, optical disks (MOs (Magnetic Optical Discs)/MDs (Mini Discs)/DVDs (Digital Versatile Discs)), IC (Integrated Circuit) cards (including memory cards), SD cards, optical cards, mask ROMs, EPROMs (Electronically Programmable Read-Only Memory), EEPROMs (Electronically Erasable Programmable Read-Only Memory), flash ROMs, and other semiconductor memories.

The term "program" here refers not only to programs that can be executed directly by a CPU, but also to programs in source program format, compressed programs, encrypted programs, etc.

Computer device 1000 receives image data (Figure 8) stored on recording medium 38 provided in drone 100. Computer device 1000 performs analysis processing using the image data. For example, CPU 1 extracts an image of hatch 150 from each image data, and can identify which blade was depicted in the image of the blade acquired when the image of hatch 150 was acquired, based on the orientation of hatch 150 and the positional relationship between each blade. In this way, an inspector who is the user of computer device 1000 can easily perform a visual inspection of the blades using the images, and can also easily identify which blade was found to be damaged.

[Image data example]
Another example of image data obtained by photography using the drone 100 will be described with reference to Fig. 11. Fig. 11 is a diagram illustrating an image 1100 photographed from above the wind turbine 10. This image data is read from the drone 100 into the computer device 1000, and an image based on the image data is displayed on the monitor 8.

The image 1100 shown in FIG. 11 was captured by the drone 100 from behind the nacelle 160. The drone 100 is equipped with an inertial measurement unit 32 and a GPS receiver 34, so position information for the drone 100 at the time of capture is obtained. If the drone 100 captures the image with the first camera 18 or the second camera 20 aligned with the direction of travel of the drone 100, the direction in which the drone 100 was facing at the time of capture can be estimated. For example, if the image 1100 is a photograph of the drone 100 from directly in front of and looking downward, the orientation of the nacelle 160 in the image 1100 can be estimated. The computer device 1000 can calculate the actual orientation of the nacelle 160 based on the orientation of the drone 100 and the orientation of the nacelle 160 in the image 1100.

[Processing procedure for acquiring a still image]
The process of identifying blades from video image data acquired according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing part of the process of identifying blades from a still image of the wind turbine 10.

In step S1210, the controller 30 estimates the current orientation of the nacelle 160 from the image captured by the drone 100. This image is acquired by the drone 100 flying in the preparation stage before starting to capture images of the blades.

In step S1215, the controller 30 sets the angle of the first camera 18 located on the top of the drone 100 or the second camera 20 located on the bottom to match the shooting position of the drone 100. This setting is performed, for example, based on the shooting angles of the first camera 18 and the second camera 20 and the distance to the wind turbine 10.

In step S1220, the controller 30 uses control using the GPS receiver 34 and LiDAR sensor 24 to move the drone 100 closer to the front or side of the wind turbine 10 while maintaining the required distance (required separation) between the drone 100 and the wind turbine 10.

In step S1225, the controller 30 causes the drone 100 to hover (stop in mid-air) at the shooting position.

In step S1230, the controller 30 begins photographing the wind turbine 10 (its blades) while synchronizing the shutter timing of the two cameras (first camera 18 and second camera 20).

In step S1235, the controller 30 ends photographing of a portion of the wind turbine 10 (partial photographing).

In step S1240, the controller 30 determines whether to change the shooting position of the drone 100. This determination is made, for example, based on whether the required number of shots, calculated in advance according to the length of the blade, have been taken. If the controller 30 determines that the shooting position of the drone 100 will be changed (YES in step S1240), the controller 30 returns control to step S1215. Otherwise (NO in step S1240), the controller 30 switches control to step S1245.

In step S1245, the controller 30 determines that all images of the inspection target have been captured. The drone 100 is then returned to the ground. The still image data stored in the recording medium 38 of the drone 100 is input to the computer device 1000.

In step S1250, the CPU 1 of the computer device 1000 analyzes the image data obtained by photographing and identifies the hatch 150. For example, the CPU 1 uses AI (artificial intelligence) to analyze an image containing the hub 140 and identify the hatch 150 from the image.

In step S1255, CPU 1 labels the image captured simultaneously with the image containing the hatch image as the first blade (e.g., blade 120). For example, if blade 120 is positioned approximately vertically below hub 140, one direction of capture is for drone 100 to capture images while ascending from the tip of blade 120 (below wind turbine 10) toward hub 140. In this case, first camera 18 positioned above drone 100 constantly captures images of hub 140, while second camera 20 positioned below drone 100 captures images of part of blade 120. Therefore, in this case, the image containing the hatch image corresponds to the image captured by first camera 18. Furthermore, the image captured simultaneously with the image containing the hatch image corresponds to the image captured by second camera 20. Completion of labeling of the first blade is determined, for example, when the drone 100 rises to a position in front of the hub 140 and an image of the hub 140 is extracted from images captured by the second camera 20. Alternatively, CPU 1 determines that labeling of the first blade is complete when analysis is completed using a number of images equivalent to the number of images captured in advance, calculated based on the length of the blade, the capture range, and the distance between the drone 100 and the blade.

In step S1260, after completing labeling of the first blade, CPU 1 starts labeling of a second blade (e.g., blade 121). Labeling of the second blade is performed in the same manner as labeling of the first blade. Completion of labeling of the second blade is determined in the same manner as completion of labeling of the first blade.

In step S1265, after completing labeling of the second blade, CPU 1 starts labeling of a third blade (e.g., blade 122). Labeling of the third blade is performed in the same manner as labeling of the first or second blade. Completion of labeling of the third blade is determined in the same manner as completion of labeling of the first blade.

Note that labeling of the first, second, and third blades may be performed simultaneously by CPU 1 without separating the steps.

In step S1270, CPU 1 reconstructs an entire image of each blade using each image for which labeling has been completed. For example, CPU 1 creates an entire image by arranging each image labeled as the same blade in the order of their capture times. In this case, the same part may be included in multiple images. Note that since each image has been labeled and the captured blade has been identified, it is not necessary to reconstruct an entire image.

In step S1275, CPU 1 displays an image of each blade on monitor 8. The displayed image may be either a labeled individual image or a reconstructed overall image. An inspector can inspect for damage by looking at the image displayed on monitor 8 and checking the surface condition of each blade.

[Processing procedure for acquiring moving images]
The process of identifying blades from video data acquired according to this embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing part of the process performed to identify blades from video images of the wind turbine 10. Note that the same processes as those shown in Fig. 12 are assigned the same numbers. Therefore, the same processes will not be described repeatedly.

In step S1330, the drone 100 simultaneously begins video capture using two cameras, namely the first camera 18 and the second camera 20. The video data acquired by the first camera 18 and the video data acquired by the second camera 20 are associated with the time acquired from the GPS receiver 34.

In step S1345, the controller 30 determines that video recording of all inspection targets has been completed. The drone 100 is then returned to the ground. The video data stored in the recording medium 38 of the drone 100 is input into the computer device 1000.

In step S1350, the CPU 1 of the computer device 1000 analyzes the video data obtained by shooting and identifies the hatch 150. For example, the CPU 1 uses AI (artificial intelligence) to analyze each frame image that makes up the video data containing the hub 140, and identifies the hatch 150 from the frame image.

In step S1355, CPU 1 labels the image frame acquired simultaneously with the image frame containing the hatch image as the first blade (e.g., blade 120). For example, if blade 120 is positioned approximately vertically below hub 140, drone 100 captures video while ascending from the tip of blade 120 (below wind turbine 10) toward hub 140 as one direction of capture. In this case, first camera 18 positioned above drone 100 constantly captures images of hub 140, while second camera 20 positioned below drone 100 captures images of a portion of blade 120. Therefore, in this case, the image frame containing the hatch image corresponds to a frame constituting the video captured by first camera 18. Furthermore, image frames captured simultaneously with the image frame containing the hatch image correspond to frames constituting the video captured by second camera 20. Completion of labeling of the first blade is determined, for example, when the drone 100 rises to a position in front of the hub 140 and an image of the hub 140 is extracted from the video captured by the second camera 20.

[Processing procedure for acquiring still images and moving images]
In another aspect, moving images and still images may be acquired simultaneously. In this case, if the timestamp of a frame included in a moving image is the same as the timestamp of the still image, the frame and the mounted still image may be used to simultaneously capture images of the hub 140 and the blades.

Referring to Figure 14, we will now explain the imaging according to this other aspect. Figure 14 is a flowchart showing part of the process performed to identify a blade from video and still images. Note that the same processes as those described above are assigned the same numbers. Therefore, the same processes will not be described again.

In step S1430, drone 100 uses one of the two cameras to capture still images and the other to capture moving images. For example, drone 100 captures moving images of hub 140 and still images of blades using the other camera. The camera used to capture moving images is determined based on the shooting position and shooting direction of drone 100 (for example, the direction from the tip of the blade toward hub 140, or the opposite direction). In other aspects, drone 100 may capture still images of hub 140 and moving images of blades.

In step S1435, the controller 30 ends partial capture. That is, the drone 100 ends capturing still images and video images at the initial position. In this case, the drone 100 may pause the camera used to capture video images so that capture at the next position can be quickly achieved.

In step S1240, the controller 30 determines whether to change the shooting position. If the drone 100 determines to change the shooting position (YES in step S1240), it switches control to step S1225. If not (NO in step S1240), the controller 30 switches control to step S1445.

In step S1445, the controller 30 determines that still and moving images have been captured for all inspection targets. The drone 100 is then returned to the ground. The still image data stored in the recording medium 38 of the drone 100 is input into the computer device 1000.

In step S1450, CPU 1 of computer device 1000 analyzes the video and still image data obtained by shooting to identify hatch 150. For example, CPU 1 identifies an image frame from the video that contains the same timestamp as the timestamp of the still image data, and determines whether a portion corresponding to all or part of hatch 150 is included in either the image frame or the still image. Once CPU 1 identifies hatch 150, it distinguishes between images that do not contain hatch 150 and images that contain a blade, and begins identifying the blade (step S1355).

In this way, the drone 100 may capture images of the hub 140 and blades using one of the two cameras to capture still images and the other to capture moving images.

[Processing Procedure for Photographing the Hub 140 and Blades with One Camera]
In yet another aspect, if a single camera can capture a wide range of images, the drone 100 may capture images of the hub 140 and the blades with a single camera. In this case, each image data set includes the hub 140 and the blades, eliminating the need for processing to associate multiple image data sets.

The processing in this situation will now be described with reference to Figure 15. Figure 15 is a flowchart showing part of the processing performed when photographing the hub 140 and blades using a single camera. Note that the same processes as those described above are assigned the same numbers. Therefore, the same processes will not be described again.

In step S1515, the controller 30 adjusts the angle of the one camera used to photograph the hub 140 and blades to match the position of the drone 100. The camera used for photographing may be either the first camera 18 or the second camera 20. The camera may be selected, for example, based on the positions at which the first camera 18 and the second camera 20 are installed on the drone 100.

In step S1530, the controller 30 starts video capture using either the camera 18 or the second camera 20.

In step S1535, the controller 30 ends capturing images of the wind turbine 10 from one direction.

In step S1540, the controller 30 determines whether to change the drone's shooting position. This determination is made in the same manner as the determination in step S1240. If the controller 30 determines that the drone's shooting position will be changed (YES in step S1540), the controller 30 returns control to step S1515. Otherwise (NO in step S1540), the controller 30 switches control to step S1345.

In step S1555, the controller 30 labels the blade appearing on the left side of the screen in the same frame as the hatch as the first blade.

As described above, if a single camera has a wide angle of view and can capture images of the hub 140, portions of the blades, and the hub 140 in the same frame, the drone 100 may use a single camera to capture images of the blades and hub 140 and acquire image data of the blade surfaces. Note that the flowchart illustrated in Figure 15 shows the case where video images are captured, but the drone 100 can also acquire image data in which portions of the blades and the hub 140 are included in the same frame when a single camera continuously captures still images or when a single camera simultaneously captures continuous still images and video images.

[Example of image]
16 and 17, a description will be given of images acquired by photography using the drone 100. Fig. 16 shows an image obtained by the first camera 18 photographing the hub 140.

As shown in Figure 16, a hatch 150 is provided on the hub 140, so the blade photographed can be identified based on the positional relationship between the hatch 150 and the blade in the image.

Figure 17 shows image 1700 of the tip of blade 120 identified in step S1225. The tip contains scratches 1710 and 1720. An inspector can inspect blade 120 and determine if repairs are necessary.

As described above, a drone 100 according to one embodiment has two cameras, and while the timing of each camera's image capture is synchronized, one camera captures an image of a portion of a blade, while the other camera captures an image of a predetermined portion of the wind turbine 10, such as the hatch 150. Because the hatch 150 and a portion of a blade are always captured, image analysis after capture identifies which blade the image corresponds to based on the presence or orientation of that portion and the positional relationship between each blade and the hatch 150. As a result, even if only a portion of a blade is captured, it is possible to identify which blade's image was acquired, and therefore, if a scratch or other damage is found during a visual inspection using the image, it is easy to determine which blade the damage is.

The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not the above description, and is intended to include all modifications that are equivalent in meaning to and within the scope of the claims.

The disclosed technical concept can be used to obtain data necessary for visual inspections using drones or other unmanned aerial vehicles equipped with equipment capable of photographing two locations simultaneously.

1 CPU, 2 Mouse, 3 Keyboard, 4 RAM, 5 Hard disk, 6 Optical disk drive, 7 Communication interface, 8 Monitor, 9 CD-ROM, 10 Windmill, 11 Main body, 12 Arm, 13 Propeller motor, 14 Propeller, 15 Motor drive circuit, 16 Engine, 17 Generator, 18 First camera, 18A, 18B, 20A, 20B Camera, 19, 21 Gimbal, 20 Second camera, 22 Leg, 23 Propulsion mechanism, 24 Sensor, 30 Controller, 31 Storage device, 32 Inertial measurement unit, 33 Transmitter/receiver, 34 GPS receiver, 35 Power supply circuit, 36 Storage battery, 37 Reader/writer, 38 Recording medium, 100, 100A, 100B, 100C Drone, 120, 121, 122 Blade, 130 Tower, 140 hub, 150 hatch, 160 nacelle, 180 foundation, 210, 220, 230, 240 route, 700, 1100, 1500 images, 810, 811, 820, 821 image data, 1000 computer equipment.

Claims (14)

using one or more cameras mounted on an unmanned aerial vehicle to simultaneously photograph a pre-specified portion of a hub of a wind turbine and a portion of one of a plurality of blades attached to the hub multiple times;
repeating the step of performing simultaneous imaging multiple times for the remaining blades of the plurality of blades;
A method for acquiring data for inspecting a wind turbine, comprising a step of identifying the photographed blade based on the presence or absence of the pre-identified part or the orientation of the part in an image based on image data acquired by the simultaneous photographing, and the positional relationship of the multiple blades to the pre-identified part. The step of performing simultaneous imaging a plurality of times includes:
simultaneously capturing an image of a pre-specified portion of the hub with a first camera of the one or more cameras and capturing an image of a portion of one of the blades with a second camera of the one or more cameras;
2. The method for acquiring data for inspecting a wind turbine according to claim 1, further comprising: in response to simultaneously photographing the pre-specified portion and the part, associating the image data acquired by the first camera and the image data acquired by the second camera with the time at which the photographs were taken. the step of simultaneously capturing images multiple times includes acquiring still images;
3. The method for acquiring data for inspecting a wind turbine according to claim 2, wherein the step of associating the time includes associating the time with the image data acquired by the first camera and the image data acquired by the second camera. the step of simultaneously capturing images a plurality of times includes acquiring a moving image;
3. The method for acquiring data for inspecting a wind turbine according to claim 2, wherein the step of associating the time includes associating the time with a frame included in the video data acquired by the first camera and a frame included in the video data acquired by the second camera. The method for acquiring data for inspecting a wind turbine as described in claim 2, wherein the step of taking simultaneous photographs multiple times includes a step of directing the photographing direction of the first camera toward the pre- identified part so that the pre-identified part does not leave the photographing range as the unmanned aerial vehicle moves. The cameras are provided on the upper and lower parts of the housing of the unmanned aerial vehicle,
The method for acquiring data for inspecting a wind turbine described in any one of claims 2 to 5, wherein the step of simultaneously photographing includes, when the blade to be photographed is above the hub, having the camera provided at the upper part act as the second camera to photograph a part of one of the blades, and having the camera provided at the lower part act as the first camera to photograph the pre-specified part. The cameras are provided on the upper and lower parts of the housing of the unmanned aerial vehicle,
The method for acquiring data for inspecting a wind turbine described in any one of claims 2 to 5, wherein the step of simultaneously photographing includes, when the blade to be photographed is below the hub, using the camera provided at the lower part as the first camera to photograph a part of one of the blades, and using the camera provided at the upper part as the second camera to photograph the pre-specified part. 8. The method for acquiring data for inspecting a wind turbine according to claim 1, further comprising a step of identifying the blade to which the image of the part of one blade acquired at the time the image including the pre-identified part is acquired belongs. The step of identifying includes:
analyzing a plurality of said images to extract said pre-identified region from each said image;
and labeling blades included in the image of the portion of one blade acquired at the timing when each of the images was acquired, based on blade identification information that is pre-specified based on a positional relationship between the pre- specified portion and the plurality of blades. A method for acquiring data for inspecting a wind turbine as described in any one of claims 1 to 9, wherein the step of taking simultaneous photographs multiple times includes taking the simultaneous photographs while hovering the unmanned aerial vehicle while the blades are rotating. A method for acquiring data for inspecting a wind turbine as described in any one of claims 1 to 9, wherein the step of taking simultaneous photographs multiple times includes taking the simultaneous photographs while moving the unmanned aerial vehicle while the blades are stationary. Moving the unmanned aerial vehicle includes:
moving the blade tip to the hub; or
The method for acquiring data for inspecting a wind turbine according to claim 11 , further comprising moving the unmanned aerial vehicle from the hub towards the tip of a blade to be photographed. A program that causes a computer to execute the acquisition method described in any one of claims 1 to 12. An unmanned aerial vehicle,
a memory storing the program according to claim 13 ;
one or more cameras disposed on the unmanned aerial vehicle;
and a control device that executes the program to control the unmanned aerial vehicle.