ES2585251B2 - INVISIBLE DRON - Google Patents

Abstract

Drone that is characterized in that each of the external parts is coated or surrounded by a screen of a uniform reflectivity and color material. And it includes an image capture device that makes other drones, equally coated, invisible in the captured photographs. # This device applies a new method that allows to distinguish the area that each drone occupies in the image, for later elimination by means of techniques of thresholding by color and image restoration.

Description

INVISIBLE DRON

TECHNICAL SECTOR The present invention falls within the technical area of unmanned aerial vehicles and their methods of use. Specifically, that which concerns unmanned aerial vehicles that are used for the acquisition of images and video.

BACKGROUND OF THE INVENTION Drones or unmanned aerial vehicles (VANT), whose acronym in English correspond to UAV (Unmanned Aircraft Vehicle), have evolved rapidly in recent years. These types of unmanned aircraft have gone from exclusively military purposes to widely used in all types of civilian tasks. The diversity of these tasks has resulted in the specialization of the designs of these aircraft as well as their control systems, with the consequent variety of shapes, sizes, configurations and features adapted to each of the tasks.

Typically, it is usually known by the acronym UAS (Unmanned Aircraft System) to the assembly that includes the aircraft, the station or control system from where it is operated and (optionally) the unit that is responsible for data transmission.

Drones can be considered as platforms that carry inside a set of sensors and are intended to obtain geospatial data. This data can guide your piloting, to the point that, depending on the sophistication of the control system, the drone can operate completely autonomously under the supervision of a pilot, determining in real time the path to be traveled.

However, the most popular models are usually controlled remotely from a ground station by an operator. Therefore, the term "unmanned" may be the source of some confusion, since while it is true that no crew is traveling inside the vehicle, it is also true that there is continuous communication between the UAV and ground operators, Either pilots, controllers or any other type of operator related to the monitoring of the aircraft.

The lack of regulation or its laxity in some countries, as in the case of the United States of America, has favored the popularization of this type of technology and the creation of new market niches. For example, the US FAA (Federal Aviation Administration) allows fans to use small aircraft within the operator's field of vision and at heights below 400 feet in areas that are not densely populated.

On the other hand, the audiovisual commercial sector has shown a special interest in the use of such devices, since unmanned aircraft can be used to make video and photo plans that would otherwise require a helicopter or a manned aircraft, that supposes a strong reduction of costs.

This technology is especially interesting in the recording of sporting events. A clear example of its application's success was the 2014 Winter Olympics in Sochi, where drones with cameras were used to film ski and snowboard events. In both cases, this type of devices allowed to take images of great spectacularity when approaching the athletes, and to provide greater flexibility than the systems of cameras placed in suspended cables.

Given its relative low cost, an amateur can record himself during the practice of any outdoor sport if the UAS is able to follow a visual mark or a radio signal; concentrating on performing sports, without the need for an operator or pilot to control the flight path. Products that meet these characteristics can be found on the market such as AirDog [http: //www.airdog.coml] of Helico Aerospace Industries US LLC, Lily [http://www.lily.camera/] of Lily Robotics, Inc., HEXO [https: //hexoplus.com~ of Squadrone System or PlexiDrone [https://www.indiegogo.com/projects/plexidrone-making-aerial-photography-a-breeze#1] DreamQii.

This fact can be generalized and it is easy to imagine sporting events in which several fans or even television networks simultaneously use this technology to capture and transmit live sequences of great visual richness.

The most sophisticated UAS have mechanisms to avoid colliding with vegetation

or any other type of obstacle in your path, including other drones even if they are not piloted in a coordinated manner.

However, when more than one camera drone is used to record scenes on the same group of actors or athletes, it is especially difficult to obtain good frames without these recording instruments being present in the field of vision. So, in good part of the recordings made with each of the drones, in addition to recording the people of interest, they also totally or partially capture other drones, which can distract the viewer's attention.

Since the drones in this case are auxiliary elements that help to make the recordings, many television or filmmakers discard the shots in which they are visible, since they are not interested in appearing in the composition. This situation is all the more common the higher the number of unmanned aerial vehicles with cameras in the recording area.

This problem is almost inevitable, even using the most modern trajectory planning systems such as the one recently published by Niels Joubert, Mike Roberts, Anh Truong, Floraine Berthouzoz, and Pal Hanrahan in SIGGRAPH Asia 2015, entitled An Interactive Tool for Designing Ouadrotor Camera Shots . Different camera operators and autonomous systems can choose frames from angles in which other drones, or their reflection, are shown in the scene.

This situation is inconvenient. The presence of these types of devices, to which viewers are not yet accustomed, tends to distract viewers' attention. Its elimination through postproduction tools requires the manual work of qualified professionals. Thus, the cost in time precludes its realization in live broadcasts.

It would therefore be desirable for drones to fulfill their function without taking center stage in the photographs.

The applicant of the present invention is unaware of the existence of a background that satisfactorily solves the problem presented.

The present invention offers a solution to this technical problem, both in the case of live productions and in recordings that are transmitted in deferred. This solution allows to detect and distinguish the portion of the image in which the drones appear in the field of vision, so that it can later be eliminated using digital image restoration techniques (inpainting in English). Thus, the images are obtained so that the content of interest is preserved without elements that divert the

viewer's attention, by eliminating the presence of drains that meet the claimed characteristics.

EXPLANATION OF THE INVENTION Drones are tools that allow camera operators to obtain very spectacular images. The sophistication of the control systems (UAS) already makes it possible for your trajectory to be calculated in an unsupervised manner and the taking of video sequences is automated at a very low cost. These factors favor the use of a large number of these devices in all kinds of outdoor events, such as in extreme sports competitions.

As already indicated, the presence of swarms of unmanned vehicles that try to capture images of an event simultaneously, entails the problem of the visibility of these devices in the recorded sequences.

The only way to prevent a drone from appearing in the field of view of another drone's camera is to restrict the space in which they can fly around points of interest. Which in turn makes it impossible to take simultaneous images from certain angles.

In the case of film production, the sequences must be repeated with the consequent investment in personnel and time for their realization. But in the case of a live event, this restriction is especially inconvenient, since the scenes cannot be repeated and the limitation in the positions of the drones implies the loss of those planes.

Instead of limiting the position of the drones, the present invention proposes to make the recordings in a natural way according to the criteria of the producer and subsequently eliminate the presence of the drones in the photographs or video sequences using digital image processing techniques.

Image reconstruction algorithms for the removal of objects (in English inpainting) require that previously distinguish those parts of the image to be removed. Its automatic determination is still an open field in research, since - in general - the objects to be removed depend on the preferences of the user and the context offered by the composition in the photograph. However, in the case of images taken from multiple perspectives with a set of drones, the

Objects to eliminate are usually the drones that carry the cameras from which the images have been taken. Our invention facilitates that an automated image processing system can differentiate between the drains and the rest of the scene in the captured image, by means of a simple color thresholding, so that it can be segmented and eliminated by inpainting techniques.

For this, it is proposed that the drone be covered with a material with a uniform reflectivity, of a uniform and characteristic color, unlike the plastic or metallic finish that the drones marketed so far usually present. A material with uniform reflectivity causes the light incident on the surface to be reflected in all angles equally (so that materials that have this type of interaction with light are usually known as lambertians), which prevents reflections from occurring or shine on its surface.

The coating can be done in two ways: directly on the fuselage and the wings of the drone or by placing a screen around it made of a material that meets the described properties.

This screen panoramicly surrounds the drone, so that it is not visible from the point of view of other drones. The screen has holes to not obstruct the camera's vision and to let the air flow through it. Although the screen can affect the aerodynamics of the device, its geometry and lightness do not alter the flight capacity and it has the advantage of exercising additional protection against possible blows or forced landings. The screen allows the use of the present invention by adapting the recording equipment and drones already existing in the market.

In this explanation both alternatives are proposed simultaneously and interchangeably, being essentially part of the same invention.

As in any material, on the external surface of the drone there is a smooth variation in irradiance (flow of light energy per unit area), due to the angle of incidence of light and the curvature of its surface, which results in a shadow on the surface of the fuselage and the screen. For the thresholding process to identify the area of the image corresponding to the drone by selecting a set of colors close to the chosen one, it is convenient that the segmentation be done in terms of the chromaticity of said color in colorimetric spaces such as CIE 1976 L * a * b or the CIE 1976 L · u · v, in which the intensity of the light is specified with an independent component (which can be ignored to increase the robustness against the soft variation of irradiance). This strategy is similar to the one applied in the choma-key in the audiovisual industry.

If any element of the scene has the same color as the drone coating, both would be removed from the image. This unwanted effect is the main limitation of thresholding and segmentation by color. Therefore, although it would be feasible to cover the drone of any uniform matte color, it is preferable that the color is difficult to find in the scenarios in which the video shots are to be taken.

Since, regardless of race, the color of human skin pigmentation is eminently red, it is preferable that the material that covers the drone has a uniform color as far away as possible from red in the mentioned calorimetric spaces. As, for example, certain shades of green or blue that are not usually found in natural or urban environments. An example of this type of colors is the light green that is formed by taking only the green channel and a high level of luminosity in the RGB calorimetric space with normalized coordinates [0,1,0], which stands out for "artificiality" as it is not common in the mentioned environments.

Many of the unmanned aerial vehicles used for video recording resemble helicopters with multiple rotors for their support and propulsion in the air. These types of designs allow them to use smaller blades, which have several advantages: they provide the possibility of raising the flight with less kinetic energy, that the pilot has greater control in the maneuvers and, more importantly, reduces the possible damage in the case of a collision with a person. In order to protect people and the vehicle if both collide, the blades are usually surrounded by fuselage bars that, as a shield, avoid part of the possible blows without reducing the volume of air that must move. These structures must be removed from the images, as well as the possible bias in the color produced by the blade on the stage objects that are seen through it when it is in motion.

Another aspect to keep in mind is that the lens cannot be colored, because the paint would block the passage of light and images could not be captured. But it must be removed from the image just like the rest of the drone that appears in the field of view.

Therefore, since the fuselage, the wings and the screen usually surround both the recesses of the blades and the objective when viewed in the images taken from another vehicle, the proposed system detects and marks not only those areas that have the chosen color but also those that are completely surrounded by areas that have the chosen color, or that are framed by those areas and the border of the field of vision.

Thus, a set of drones whose pieces are covered by a uniform colored Lambertian material can be used for the proposed purpose, in combination with digital image capture devices in which the method comprising the following steps is carried out:

one.

Capturing the image with a digital camera

2.

The generation of masks as a result of the process of thresholding the image with respect to the uniform color of the material that covers the drains, in which those pixels whose color coincides or has a small variance in chrominance with respect to the chosen color are individually marked.

3.

The grouping and labeling of the pixels marked in adjacent regions, for easier identification.

Four.

The determination of the contours of the labeled regions.

5.

In order to also mark the objective lens and the cavities where the drone blades are located, it is calculated whether the same region is bounded by several contours and if they contain each other. In which case the one with the largest area (corresponding to the outer contour) is chosen and all the pixels it contains are labeled as belonging to the main mask in that area.

6.

The search for the labeled regions of the mask in the upper, lower, right and left limits of the image.

7.

The selection of labeled regions located on the edges of the image.

8.

Step 5 is repeated in the previously selected regions, considering that the border of the image acts by delimiting contours if the length necessary for this is less than a certain threshold (such as one sixth of the length of the smaller side of the image) .

9.

The application of an image restoration algorithm for the removal of objects (inpainting) on the selected portions of the mask through the labeled regions, so that the portions of selected masks are filled with information from the rest of the image (or from other images). There are numerous known algorithms that allow you to replace portions of the image with the information present in

the rest of the image, such as the techniques protected by US patents US 6987520, US 7551181, US 7755645 and US 7840086. At this stage any of the mentioned techniques or other more sophisticated techniques that take advantage of temporal coherence in the sequences could be used. of video, on the areas marked with the masks.

At the end of the stages, no other drones will appear in the resulting image. Since instead a set of patterns will be drawn that replace them visually. The better the image restoration algorithm chosen, the more realistic the result of such substitution, which makes each of the drones disappear.

BRIEF DESCRIPTION OF THE DRAWINGS To complement the description that is being made and in order to help a better understanding of the characteristics of the invention, a set of drawings is attached as an integral part of said description, where it is illustrative and not limiting , the following has been represented:

Figure 1.- Shows an artistic representation of a photograph taken from a drone in a sporting event in which a runner and other drones covered in a lambertian material of uniform color according to the claims appear in the frame. In a dashed line the field of vision of each of the image capture devices is symbolized, being exposed to each other in the shots of the race and hiding part of the scene of interest.

Figure 2.- It shows the pixels marked after the thresholding by color in the artistic representation of the photograph of figure 1. The holes left by the lenses (5) and the spaces in which the blades (6) of the drones

Figure 3.- Shows the masks (7) generated in the artistic representation of the photograph of figure 1, after the application of step 5 of the described method.

Figure 4.- Shows the result of eliminating the areas covered by the masks (7) in the artistic representation of the photograph of figure 1, before carrying out step 9 of the described method. Without the application of image restoration techniques the result can be as disturbing to the viewer as the presence of the drones, by distracting their attention from the sporting event.

Figure 5.- Shows an artistic representation of the result of applying the proposed method completely to a photograph when the drains are covered by a uniformly colored Lambertian material.

Figure 6.- Shows a perspective view of a drone that has been incorporated into a screen that surrounds it panoramicly, of Lambertian material of uniform color with an aperture (12) for capturing images with the objective.

PREFERRED EMBODIMENT OF THE INVENTION Figure 6 shows a perspective view of a particular embodiment of the invention. In this case the drone (11), quadcopter type, has been covered panoramicly by a pan (8) that has a recess (12) that surrounds the lens (5) of the objelivo (10), which allows the taking of images without the screen blocking the field of view. No additional openings have been necessary, since the screen does not impede the passage of air for the operation of the blades. The screen is fixed to the quadcopter by means of a series of fasteners (9) in the structure that protects the propellers.

Alternatively, a similar result could have been obtained if all the outer parts of the quadcopter were covered with a reflective material and uniform color.

In this example, we chose to use a light green matte paint on the screen (8) (due to the limitations of color reproduction in the figures, it is shown in a gray tone). On the one hand, this color has been chosen because it is markedly different from the tone of human skin, and on the other hand because it is less common than blue in urban environments (the sky usually has bluish tones and blue clothes, such as jeans , is widely used in the West). In addition, digital camera sensors tend to be more sensitive to green, because the Bayer pattern that filters have on their surface dedicates more sensors to the green channel (mimicking the greater sensitivity of the human visual system to this frequency range in the visible spectrum), so less light is needed to illuminate the greenish colors.

The automated system would consist of one or several drones (1,2,3,11) whose outer surface is covered with a material of reflectivity and uniform color, in this case light green, either directly or with a screen (8). Equipped with a digital image acquisition device which in turn is composed of: -an optical sensor in the visible range, together with the set of lenses (5) necessary to focus the image

-

a data processing unit (typically a digital processor)

-

one or more memories

-

a communication unit

These last three act in a coordinated way (by means of a program that encodes the steps) on the sensor to register and store the photograph at the moment indicated by the user, and then apply the method described above.

In this particular embodiment. The color thresholding that selects the pixels that potentially correspond to the areas occupied by the drone considers as candidates those with a chrominance variance of less than 4% with respect to the chosen color. In the example of the representation of a photograph of Figure 1, the external surface of the drones would be detected and labeled as regions of the masks. However, as shown in Figure 2, the gaps (6) in which the propellers rotate reveal partially through them and, like the lenses (5) of the lenses (10), are not labeled as part of the masks through a simple thresholding by color.

For this reason, the proposed method identifies the external contours of each previously marked region and fills all the pixels inside, with the identifier corresponding to that of the main mask in the affected area.

On some occasions, the hollow of a blade or a portion of a lens is located at the edge of the image, so that from the point of view of the drone that takes the image, the hole of the blade or the lens is no longer surrounded for the uniform color and reflectivity material, so that the step described above would not label these regions for later replacement in the image. In order to incorporate these elements in the labeling of the masks, it is considered that the border of the image can act as part of the outer contour of the same mask if the length necessary to connect two different parts of it is less than one sixth of the length of the side of the image (the smallest in the case of being rectangular). And the set of pixels that is covered by the contour so extended is labeled, for later treatment with the inpainting algorithm, as can be seen in Figure 3.

Finally, the image restoration algorithm applied is that published in the article "Object Removal by Exemplar-Based Inpainting" by A. Criminisi, P. Perez and K. Toyama. Algorithm that has been mentioned above, since it is protected with US patent 6987520 with the title "Image regian filling by exemplar-based inpainting".

5 Figure 5 shows the image that would be obtained as a result of the application of the method by means of the automated system proposed in this invention with invisible drones.

Once the nature of the present invention has been sufficiently described, as well as a preferred embodiment, it only remains to be added that said invention may suffer

10 certain variations in form, materials, parameters and applicable image restoration algorithms, as long as said alterations do not substantially vary the characteristics claimed below.

Claims (4)

one.

Method for the elimination by a drone of the images of other drones present in the images captured by it, characterized in that the method comprises the following steps:

2.

Drone, for the implementation of the method of elimination in an image according to the preceding claim, characterized by being coated with a uniform reflectivity and color material, and being provided with a digital image acquisition device in which drones are eliminated by a color similarity criterion that includes:

to.

image capture with a digital camera;

b.

the generation of masks due to image thresholding with respect to

a uniform color of the material that covers or surrounds the drone, in which they are marked

individually those pixels whose color has

a variance in chrominance

less than 4% with respect to the chosen color;

C.

the grouping and labeling of the pixels marked in adjacent regions;

d.

the determination of the external contours of the labeled regions;

and.

the group Y labelled from the pixels in he inside from the contours

delimited;

F.

the search for the tagged regions of the mask in the upper limits,

bottom, right and left of the image;

g.

the selection of mask portions located at the edges of the image;

h.

the determination of new external contours of the same selected region

when different portions of it are connected, when considering the border of the

image as part of said contour, provided the length of the extension is

less than one sixth of the smaller side of the image;

i.

the grouping and labeling of the pixels inside the new contours

delimited;

j.

the application of a inpainting algorithm on mask portions

labeled in the image.

-

an image capture system (2);

-

a processor; -one or several memories in communication with the processor and the image acquisition system, which store an application executable by the processor; wherein the application can, in execution, perform the method according to claim 1.

3. Drone, according to the preceding claim, characterized in that the coating of 5 uniform reflectivity and color of each of its external parts, except the lens of the capture device, is applied directly to them. 4. Drone according to the preceding claim, characterized in that the uniform reflectivity and color coating is carried out by means of a screen (8) surrounding the total drone or partially and has at least one opening (12).