CN111626320B

CN111626320B - Method for detecting objects using two neural networks

Info

Publication number: CN111626320B
Application number: CN202010128108.4A
Authority: CN
Inventors: S·克勒
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2019-02-28
Filing date: 2020-02-28
Publication date: 2025-10-10
Anticipated expiration: 2040-02-28
Also published as: DE102019202751A1; CN111626320A

Abstract

The invention relates to a method and a device for detecting objects, wherein objects are detected in visual data detected by means of sensors, wherein two neural networks are used for object detection, and wherein the object detection comprises the method steps of operating a first neural network for detecting a geometry in the visual data, wherein the detected visual data is used as an input for the first neural network, and operating a second neural network for detecting objects in the visual data, wherein not only the detected visual data but also the geometry detected by means of the first neural network is used as an input for the second neural network.

Description

Method for detecting objects by means of two neural networks

Technical Field

The invention relates to a method and a device for detecting objects, wherein objects are detected in visual data detected by means of sensors, wherein two neural networks are used for object detection, and wherein the object detection comprises the following method steps:

Operating a first neural network to identify a geometry in the visual data,

Wherein the detected visual data is used as an input for the first neural network;

Operating a second neural network to identify an object in the visual data,

Wherein not only the detected visual data but also the geometry recognized by means of the first neural network are used as input variables for the second neural network.

Background

It is known from the prior art to automatically identify objects, for example, in autonomous driving areas by machine learning (MACHINE LEARNING) via a neural network (i.e., deep neural network, english: DNN, deep neural networks). When recognizing objects by machine learning, it is unclear what the DNN specifically learns, i.e., how the DNN recognizes the object. However, the recognition rate is obviously superior to that of a handwriting programming algorithm.

However, the risk in the detection of objects by DNN is that (for example by means of ADVERSARIAL PERTUBATIONS) the object is attacked in a targeted manner. In the attack, the actual object is maliciously manipulated. That is, no intervening database or DNN learning process is required. For example, a real object (e.g., a traffic sign) is intentionally added with graphical information (e.g., by a specifically formed tag). The information is also detected and evaluated when detecting the object. The additional information is formed such that it influences the evaluation. The identification of the object is influenced based on the transmitted additional information. It is thus possible that, due to the added information, the maliciously operated object is identified as a further object.

Disclosure of Invention

In contrast, the method according to the invention advantageously achieves that an object can be accurately identified even in the case of a malicious manipulation of the object. This is achieved according to the invention by the features given in the independent claims. Further embodiments of the invention are the subject matter of the dependent claims.

According to the invention, a method for detecting objects is provided, wherein objects are detected in visual data detected by means of sensors, wherein two neural networks are used for object detection, and wherein the object detection comprises the following method steps:

Operating a first neural network to identify a geometry in the visual data,

Wherein the detected visual data is used as input for a first neural network, operating a second neural network to identify an object in the visual data,

Wherein not only the detected visual data but also the geometry recognized by means of the first neural network are used as input for the second neural network.

In this sense, the core of the invention is to extend the current DNN-based recognition with one further DNN, which is trained only for the recognition of geometric basic shapes (e.g. horizontal, vertical and oblique straight lines, circles, N-polygons). The geometry information explicitly influences the decision of the original DNN together. That is, it is proposed to expand the recognition of objects by means of DNN by means of specific geometric information. This makes the algorithm more stable against attacks by, for example, maliciously manipulating objects. This advantageously results in a higher possible DNN recognition rate than heretofore possible by geometric evaluation.

That is, a method for detecting an object is provided. For this purpose, visual data of the surroundings of the motor vehicle are detected, for example, by means of sensors. An object is detected in the detected data by means of a data processing unit. In evaluating visual data, two trained neural networks are used. The first neural network is trained to identify geometry. The visual data is used as an input layer. The second neural network is trained for identifying objects, wherein the identified geometry is used as an input layer in addition to the visual data. Recognition of an object should be understood not only as recognition of the object but also as classification of the object. A sensor for detecting visual data is to be understood as meaning, in particular, a video camera of a motor vehicle.

In an advantageous embodiment, the method is used for operating a driving assistance system of a motor vehicle. A driving assistance system is understood to be any system which performs a driving function and/or assists the driver in guiding the vehicle. For example, the corresponding driving assistance system enables automatic recognition of objects to enable automatic driving. Of course, partially automated driving functions can also be assisted. Other driver-assist functions, such as identifying and displaying traffic signs, may alternatively be assisted.

In an advantageous embodiment of the method, visual data of the surroundings of the motor vehicle are detected by means of a sensor. This is understood to mean that the motor vehicle comprises a camera by means of which at least a part of the surroundings of the motor vehicle is recorded. The data thus generated are then evaluated by the method according to the invention for identifying objects.

In one possible embodiment of the method, the first neural network is trained in one method step for identifying simple geometries.

DNNs were first trained to recognize simple geometries. For example, the DNNs are trained to identify typical shapes of traffic signs. In the example of the priority travel section sign, this is, for example, four oblique lines and a square as the entire shape. As a learning basis for DNN, the large data acquisition of differently photographed traffic signs requires the combination of individual information about the geometry. The recognition of the geometry can then be extended to the spatial shape and, if necessary, to the motion model. A further DNN is then trained not only for processing the image information of the visual data, but also for learning the geometric information of the geometric DNN described above as a further input variable. Object recognition can thus be optimized. Methods for DNN and DNN software learning may invoke established and available methods, such as Tensor Flow Framework, in order to optimize resource usage. Of course, learning methods and ways specifically designed for the application case can also be developed and used in order to achieve the best possible results.

In a preferred embodiment of the method, the first neural network is trained in a method step for identifying the spatial geometry.

It is understood that in one of the aforementioned method steps, the first neural network is trained not only for identifying simple geometric models, for example straight lines, but also for identifying complex geometric shapes, for example body shapes, of the motor vehicle. The trained geometry DNN is then used to identify and classify the geometry in the visual data. The information thus determined is transmitted as additional information to the further DNN in order to optimally identify the object in the visual data.

In an alternative further development of the method, the first neural network is trained in a method step for identifying the movement model.

It is understood that in the aforementioned method steps, the first neural network is trained not only for identifying simple geometric models, but also for identifying complex movement models, such as the movement patterns of pedestrians, persons or animals on bicycles. A motion model is understood here as a dynamic change of a complex geometry. The person is here, for example, a complex geometry, and the movement of the person is a dynamic change of the geometry. The trained motion DNN is then used to identify and classify the motion in the visual data. The information thus determined is transmitted as additional information to the further DNN in order to optimally identify the object in the visual data.

In an advantageous embodiment of the method, the method is designed to reduce the influence of disturbing content in the detected visual data when detecting objects.

This is understood to mean that a method for detecting objects is used. Advantageously, the method steps are designed such that the effects of visually malicious operations are reduced or completely avoided. This is understood to mean that an exact recognition of maliciously operated objects is increased or improved. One measure for this is for example RSR (Ratio of Successful Recognition, successful recognition rate). What may be considered as such malicious operations, also called attacks, are, for example, resistant interference, generalized interference, random noise, etc. That is, the method achieves the effect of reducing the attack or suppressing the attack attempt in the visual data as best as possible.

In one possible embodiment of the method, the method is designed for identifying traffic signs. This is understood to mean that the method is used in the identification of traffic signs. By means of the method, an optimized classification can be achieved also in the case of maliciously operated traffic signs.

The solution presented here also proposes a device designed to implement, control or carry out the steps of a variant of the method described here in a corresponding instrument. The object of the invention can also be achieved quickly and efficiently by this embodiment variant of the invention in the form of a device.

The device is understood here to mean an electronic device which processes the sensor signals and outputs control and/or data signals as a function of the sensor signals. The device may have an interface, which may be designed in hardware and/or in software. In a hardware-based embodiment, the interface may be, for example, part of a so-called ASIC system, which contains the different functions of the device. However, it is also possible for the interface to be a self-contained, integrated circuit or to be formed at least in part from discrete components. In a software-based embodiment, the interface can be a software module, which is present next to other software modules, for example on a microcontroller.

An apparatus is therefore provided, which is provided for carrying out the method. The device can be understood as a correspondingly designed data processing unit. Advantageously, the device is understood to be a data processing unit with two DNNs. The device is understood in particular to be a data processing unit having a geometry DNN and an object DNN.

The invention also relates to a computer program product or a computer program having a program code which can be stored on a machine-readable carrier or a storage medium, for example a semiconductor memory, a hardware memory or an optical memory, and which is used to execute, carry out and/or control a method according to one of the embodiments described above, in particular when the program product or the program is implemented on a computer or a device.

Drawings

It is to be noted that the individual features mentioned in the description can be combined with one another in any technically meaningful way and that further embodiments of the invention are shown. Further features and applications of the invention result from the description of embodiments according to the accompanying drawings.

In the accompanying drawings:

FIG. 1 shows the method steps of an exemplary embodiment, and

Fig. 2 shows a view of an exemplary application.

Detailed Description

The method steps of an embodiment of the invention are shown in fig. 1. In this case, in a first step S1, visual data of the surroundings of the motor vehicle are detected by means of a camera. In a next step S2, a preprocessing of the visual data is performed. The detection of the object in the visual data takes place in the next two steps. Visual data is first provided as an Input-Layer into a first neural network. In step S3, a geometry is detected in the visual data by means of the first neural network and classified. The information thus obtained is then provided as an input into a second neural network. Additionally, the detected visual data is likewise provided as a further input into the second neural network. In step S4, objects are detected in the visual data by means of the second neural network and classified. Next, in step S5, a specific action is performed on the basis of the detected object. For example, an automatically traveling vehicle is parked at the identified parking sign.

An exemplary application is shown in fig. 2. Here, a motor vehicle 1 is shown. The motor vehicle is designed as an automatic traveling bus. The motor vehicle 1 comprises, among other sensors, a camera as sensor 2 for detecting visual data of the surroundings 6. The detection area of the camera is shown by means of a dashed line. The visual data detected by means of the sensor 2 are processed in a data processing unit 3. The data processing unit 3 here comprises a first neural network 4 and a second neural network 5. A traffic sign 7, a parking sign is in the detection area of the sensor 2. The traffic sign 7 has an element 8 which acts as a disturbance or error-containing content in the detected visual data, the so-called resistance disturbance. Such an element 8 is formed in such a way and in such a way that it thereby strongly influences the neural network-based evaluation. This may, for example, result in a parking sign not being recognized as the parking sign or being recognized as a further traffic sign in a wrong manner.

Claims

1. A method for detecting objects, wherein objects (7) are detected in visual data acquired by means of a sensor (2), wherein two neural networks (4, 5) are used for object detection, and the object detection comprises the following method steps:

- running a first neural network (4) to identify geometric shapes in said visual data,

wherein the detected visual data is used as input for the first neural network (4);

- running a second neural network (5) to identify objects (7) in said visual data,

In this case, both the detected visual data and the geometric shapes identified by means of the first neural network (4) are used as input variables for the second neural network,

The first neural network is trained only for the recognition of basic geometric shapes.

The method is designed to reduce the influence of interference content (8) in detected visual data when detecting an object in the event that the object is maliciously manipulated, wherein the interference content is an adversarial perturbation.

2. The method according to claim 1, wherein visual data of the surroundings (6) of the motor vehicle (1) are detected by means of the sensor (2).

3. The method according to claim 1 or 2, wherein the first neural network (4) is trained in one method step for recognizing simple geometric shapes.

4. The method according to claim 1 or 2, wherein the first neural network (4) is trained in one method step for recognizing spatial geometric shapes.

5. The method according to claim 1 or 2, wherein the first neural network (4) is trained in one method step for recognizing movement patterns.

6. The method according to claim 1 or 2, designed for recognizing traffic signs (7).

7. A device (2, 3, 4, 5) configured to carry out the method according to any one of claims 1 to 6.

8. A computer program product comprising a program code which, when executed by a computer, causes the computer to perform the method according to any one of claims 1 to 6.

9 . A machine-readable storage medium having a program code stored thereon, wherein when the program code is executed by a computer, the program code causes the computer to execute the method according to claim 1 .