JP7527482B2 - OBJECT DETECTION DEVICE AND OBJECT DETECTION METHOD - Google Patents

Description

The present disclosure relates to an object detection device and an object detection method for detecting an object.

An object detection device mounted on a vehicle quickly detects people, obstacles, and other objects that exist in the environment in which the vehicle is used. Data on objects detected by this object detection device is used for vehicle control, warning notifications, and other purposes, making it possible to drive the vehicle safely.

Object detection devices use sensors such as radar, cameras, lidar (LIDAR: Light Detection And Ranging), and ultrasonic sensors. In recent years, with the widespread use of various sensors, fusion-type object detection devices that improve performance by combining multiple types of sensors have come into widespread use.

The object detection device described in Patent Document 1 realizes a fusion-type object detection device using a radar and a camera. The object detection device described in Patent Document 1 outputs object detection data based on object position data detected using the radar and object position data detected using the camera.

International Publication No. 2010/119860

However, the technology in Patent Document 1 above uses a high-performance camera to detect object position data in order to detect objects with high accuracy, which creates the problem of high manufacturing costs for the object detection device.

The present disclosure has been made in consideration of the above, and aims to obtain an object detection device that can detect objects with high accuracy and at low manufacturing costs.

In order to solve the above-mentioned problems and achieve the object, the object detection device of the present disclosure includes a radar that emits electromagnetic waves to an object and receives a reflected signal from the object, a signal processor that calculates radar position data indicating the position of the object and radar speed data indicating the speed of the object based on the reflected signal, a camera that captures an image of the object to obtain image information of the object, and an image processor that calculates camera speed data indicating the speed of the object based on the image information. The object detection device of the present disclosure also includes a fusion processor that outputs the radar position data and radar speed data in a first frame to an external device as first detected position data indicating the position of the object in the first frame and first detected speed data indicating the speed of the object in the first frame. The fusion processor has a data storage device that stores the first detected position data, and when the radar position data and radar speed data are lost in a second frame that is a frame after the first frame, the fusion processor generates second detected position data indicating the position of the object in the second frame and second detected speed data indicating the speed of the object in the second frame based on the first detected position data and the camera speed data acquired in the second frame, and outputs them to an external device.

The object detection device disclosed herein has the advantage of being able to detect objects with high accuracy and low manufacturing costs.

FIG. 1 is a diagram showing a configuration of an object detection device according to a first embodiment; 1 is a flowchart showing an object detection process performed by an object detection device according to a first embodiment. FIG. 1 shows a configuration of an object detection device according to a second embodiment. 11 is a flowchart showing an object detection process performed by the object detection device according to the second embodiment. FIG. 1 is a diagram showing an example of the configuration of a processing circuit provided in an object detection device according to the first and second embodiments, in the case where the processing circuit is realized by a processor and a memory; FIG. 1 is a diagram showing an example of a processing circuit included in an object detection device according to the first and second embodiments, in the case where the processing circuit is configured using dedicated hardware;

Below, the object detection device and object detection method relating to the embodiments of the present disclosure are described in detail with reference to the drawings.

Embodiment 1.
FIG. 1 is a diagram showing a configuration of an object detection device according to a first embodiment. The object detection device 100A is a fusion-type object detection device that combines multiple types of sensors, and calculates object detection data based on data obtained from the multiple types of sensors. When object detection data is lost due to the usage environment, the object detection device 100A estimates the position, speed, etc. of the object using the detection data obtained in the previous frame. The detection data output by the object detection device 100A is used for vehicle control, etc.

The object detection device 100A includes a radar 1, a camera 2, a signal processor 3, an image processor 4A, and a fusion processor 5A. The fusion processor 5A includes an identity determination unit 6A, a detection data transfer unit 10, and a data storage unit 11A. The identity determination unit 6A includes an identity determiner 7A, a disappearance determiner 8, and a disappearance extrapolator 9A.

The radar 1 emits electromagnetic waves to objects such as people and obstacles present in the environment in which the object detection device 100A is placed, and receives reflected signals from the objects. When used on a vehicle, the radar 1 generally uses the FMCW (Frequency Modulated Continuous Wave) method or the FCM (Fast Chirp Modulation) method. The radar 1 is composed of, for example, high-frequency semiconductor components, power semiconductor components, substrates, quartz devices, chip components, antennas, etc.

The signal processor 3 processes the reflected signal (received signal) sent from the radar 1 to detect the position and relative speed of the object. The signal processor 3 sends position data indicating the position of the object calculated based on the received signal from the radar 1 to the fusion processor 5A as radar position data. The signal processor 3 also sends speed data indicating the speed of the object calculated based on the received signal from the radar 1 to the fusion processor 5A as radar speed data. The signal processor 3 is composed of, for example, an MCU (Micro Control Unit), a CPU (Central Processing Unit), etc.

Camera 2 captures an image of an object to obtain image information of the object. Camera 2 is composed of components such as a lens, a holder, a CMOS (Complementary Metal Oxide Semiconductor) sensor, power semiconductor components, and a quartz device. Camera 2 may be a low-performance camera that does not detect the position of an object.

Based on the image information acquired by the camera 2, the image processor 4A performs object recognition and calculates the relative speed of the object with respect to the object detection device 100A and the direction in which the object is located. The image processor 4A is composed of, for example, an MCU, a CPU, etc. The image processor 4A recognizes objects such as people and obstacles and detects the relative speed and direction of the object using a database of feature data obtained by machine learning or deep learning. The feature data is data that indicates the features of objects such as people and obstacles.

The image processor 4A sends object recognition data indicating that an object has been recognized to the fusion processor 5A. The object recognition data is data for distinguishing whether the object is a person or an obstacle. When distinguishing between a person and an obstacle with one bit, the image processor 4A generates, for example, a "0" for a person and a "1" for an obstacle as object recognition data. The image processor 4A also sends speed data indicating the relative speed of the object with respect to the object detection device 100A to the fusion processor 5A as camera speed data. The image processor 4A also sends orientation data indicating the orientation in which the object is located to the fusion processor 5A. The image processor 4A may be a low-performance processor that does not calculate the position of the object.

In the object detection device 100A, the signal processor 3 calculates radar position data and radar speed data on a frame-by-frame basis and sends them to the fusion processor 5A. Also, in the object detection device 100A, the image processor 4A calculates object recognition data, camera speed data, and orientation data on a frame-by-frame basis and sends them to the fusion processor 5A.

The signal processor 3 calculates radar position data and radar speed data at each specific timing. The image processor 4A also calculates object recognition data, camera speed data, and orientation data at each specific timing. If the signal processing time required by the signal processor 3 and the image processor 4A is the same, the timing at which the signal processor 3 and the image processor 4A calculate data is the same. In other words, the signal processor 3 and the image processor 4A calculate each piece of data from an object at the same time.

The timing at which signal processor 3 and image processor 4A calculate data corresponds to a frame. For example, if signal processor 3 calculates data at the nth timing (n is a natural number), the nth frame contains nth radar position data and radar speed data. Also, if image processor 4A calculates data at the nth timing, the nth frame contains nth object recognition data, camera speed data, and orientation data. The (n-1)th frame is the first frame, and the nth frame is the second frame.

The fusion processor 5A performs signal processing on the radar position data and radar speed data sent from the signal processor 3, and the object recognition data, camera speed data, and orientation data sent from the image processor 4A. The fusion processor 5A outputs the result of the signal processing as the detection result of the object detection device 100A. The detection result of the object detection device 100A is the object recognition data, speed data, and position data.

In the fusion processor 5A, the identity determiner 7A of the identity determination unit 6A executes an object identity determination process based on the radar position data, the radar speed data, the camera speed data, the orientation data, and the object recognition data. The object identity determination process is a process for determining whether the object detected by the radar 1 and the object detected by the camera 2 are the same object.

The identity determiner 7A executes an object identity determination process, and if it determines that the objects are identical, it associates the radar position data and radar speed data acquired using the radar 1 with the object recognition data and camera speed data acquired using the camera 2. The identity determiner 7A sends data that associates the radar position data, radar speed data, object recognition data, and camera speed data (hereinafter referred to as first association data) to the disappearance determiner 8 connected downstream.

The loss determiner 8 determines whether or not there is signal loss of the radar position data and radar speed data acquired using the radar 1. A state in which signal loss has occurred is a state in which the object detection device 100A was able to acquire the radar position data and radar speed data in the previously detected frame, but is temporarily unable to acquire the radar position data and radar speed data in the current detection frame. In other words, a signal loss state is a state in which the radar 1 is unable to detect an object, even though the camera 2 is able to detect the object.

When there is no signal loss, the loss determiner 8 sends the radar position data, the radar speed data, and the object recognition data to the loss extrapolator 9A. When there is a signal loss, the loss determiner 8 sends the object recognition data and the camera speed data to the loss extrapolator 9A.

When the object detection device 100A detects an object using the radar 1, it is highly likely that the desired object will be lost in clutter or multipath, making it impossible to detect the object.

The loss extrapolator 9A performs data extrapolation only when the loss determiner 8 detects data loss. When there is no signal loss, the loss extrapolator 9A sends the radar position data, radar speed data, and object recognition data sent from the loss determiner 8 to the detection data transfer unit 10 as the position data (detected position data), speed data (detected speed data), and object recognition data detected by the object detection device 100A.

When there is a signal loss, the loss extrapolator 9A generates position data and speed data for the current frame, extrapolates the data to the current frame, and transfers the data to the detection data transfer unit 10. Specifically, when there is a signal loss, the loss extrapolator 9A extrapolates the position data and speed data for the current frame based on the position data and camera speed data stored in the data storage unit 11A, and sends the data to the detection data transfer unit 10. In other words, when there is a signal loss, the loss extrapolator 9A transfers the generated position data, generated speed data, and object recognition data to the detection data transfer unit 10.

The detection data transfer unit 10 transfers the position data, speed data, and object recognition data as the current frame to an external device. An example of an external device is a vehicle control device 12 that controls the vehicle. The detection data transfer unit 10 also stores the same position data transferred to the vehicle control device 12 in the data storage device 11A. The position data stored in the data storage device 11A is read by the loss extrapolator 9A if there is a signal loss in the next frame.

The detection data transfer unit 10 always stores the position data in the current frame (the latest position data) in the data storage unit 11A, and transfers the position data, speed data, and object recognition data to the vehicle control device 12. The position data that the detection data transfer unit 10 outputs to the vehicle control device 12 is the detected position data, and the speed data that the detection data transfer unit 10 outputs to the vehicle control device 12 is the detected speed data.

When there is a signal loss in the current frame, the loss extrapolator 9A estimates the object's moving direction based on the position data of the previous detected frame stored in the data storage 11A and the camera speed data of the currently detected frame, and estimates position data corresponding to the current object position and speed data corresponding to the current object speed. The loss extrapolator 9A extrapolates the estimated position data and speed data to the current frame.

Next, the procedure for detecting an object by the object detection device 100A will be described. Figure 2 is a flowchart showing the procedure for detecting an object by the object detection device according to the first embodiment.

The object detection device 100A starts generating frames for object detection (step S1). The object detection device 100A detects the position and speed of an object using the radar 1 (step S2). Specifically, the radar 1 emits electromagnetic waves to an object, receives a reflected signal from the object, and outputs the received signal to the signal processor 3. The signal processor 3 generates radar position data indicating the position of the object and radar speed data indicating the speed of the object based on the received signal from the radar 1.

Furthermore, object detection device 100A recognizes an object using camera 2 and detects the speed and direction of the object (step S3A). Specifically, camera 2 captures an image of the object to obtain image information of the object, and outputs the image information to image processor 4A. Image processor 4A performs object recognition based on the image information obtained by camera 2, and generates camera speed data indicating the relative speed of the object with respect to object detection device 100A, and direction data indicating the direction in which the object is located. Note that object detection device 100A simultaneously performs the process of step S2 and the process of step S3A.

In the object detection device 100A, the signal processor 3 sends radar position data and radar speed data to the fusion processor 5A, and also sends object recognition data, camera speed data, and orientation data indicating that the image processor 4A has recognized an object to the fusion processor 5A.

The identity determiner 7A of the fusion processor 5A executes an object identity determination process based on the radar position data, radar speed data, camera speed data, orientation data, and object recognition data (step S4). That is, the identity determiner 7A determines whether the object detected by the radar 1 and the object detected by the camera 2 are the same object.

If the identity determiner 7A determines that the objects are the same, it generates first association data that associates the radar position data and radar speed data acquired using the radar 1 with the object recognition data and camera speed data acquired using the camera 2 and sends the first association data to the disappearance determiner 8.

The disappearance determiner 8 determines whether or not the radar detection data, which is the position data and speed data acquired by the radar 1, has been lost in the current frame (step S5). In other words, the disappearance determiner 8 determines whether or not a signal has been lost.

If radar detection data is lost (step S5, Yes), i.e., if there is signal loss, the loss determiner 8 sends the object recognition data and camera speed data to the loss extrapolator 9A.

In this case, the disappearance extrapolator 9A estimates and generates position data and speed data for the current frame based on the position data stored in the data storage device 11A and the camera speed data sent from the disappearance determiner 8. That is, the disappearance extrapolator 9A estimates the current object position and speed based on the position data detected in the previous frame and the camera speed data detected in the current frame. The estimation method is as follows.

In the frame immediately before the current frame, i.e., the (n-1)th frame, the position data detected by object detection device 100A is assumed to be (Xn-1, Yn-1). In the current frame, i.e., the nth frame, the camera velocity data detected using camera 2 is assumed to be (VXCn, VYCn). Here, X indicates the horizontal coordinate relative to object detection device 100A, and Y indicates the vertical coordinate relative to object detection device 100A. Therefore, Xn-1 indicates the position in the X direction, and Yn-1 indicates the position in the Y direction. Furthermore, VXCn indicates the velocity in the X direction, and VYCn indicates the velocity in the Y direction.

From these data, the estimated position data (Xn, Yn) of the object in the current frame is expressed by the following equations (1) and (2). Tf in equations (1) and (2) is the frame update period time of the object detection device 100A.

Xn=(Xn-1)+VXCn×Tf...(1)
Yn=(Yn-1)+VYCn×Tf...(2)

Furthermore, the estimated velocity data (VXn, VYn) of the object in the current frame is as follows:

VXn = VXCn
VYn=VYCn

The disappearance extrapolator 9A extrapolates the generated position data and speed data to the current frame. That is, the disappearance extrapolator 9A generates the disappeared radar detection data and extrapolates it to the current frame (step S6). The disappearance extrapolator 9A associates the extrapolated position data, the extrapolated speed data, and the object recognition data, and sends them to the detection data transfer unit 10 (step S7). That is, the disappearance extrapolator 9A sends the estimated position data (Xn, Yn), the estimated speed data (VXn, VYn), and the object recognition data to the detection data transfer unit 10.

On the other hand, if the radar detection data is not lost (step S5, No), i.e., if there is no signal loss, the loss determiner 8 sends the radar position data (XRn, YRn) and radar speed data (VXRn, VYRn) detected by the radar 1 to the loss extrapolator 9A.

In this case, the disappearance extrapolator 9A uses the radar position data (XRn, YRn) and radar speed data (VXRn, VYRn) detected by the radar 1 as the position data and speed data as they are. The position data (Xn, Yn) and speed data (VXn, VYn) used by the disappearance determiner 8 are as follows:

Xn = XRn
Yn = YRn
VXn = VXRn
VYn = VYRn

The loss extrapolator 9A associates the adopted position data, the adopted speed data, and the object recognition data and sends them to the detection data transfer unit 10 (step S7).

The detection data transfer unit 10 sends the position data, speed data, and object recognition data sent from the disappearance determiner 8 to the vehicle control device 12. The position data sent by the detection data transfer unit 10 to the vehicle control device 12 as the (n-1)th frame is the first detected position data, and the speed data sent by the detection data transfer unit 10 to the vehicle control device 12 as the (n-1)th frame is the first detected speed data. The position data sent by the detection data transfer unit 10 to the vehicle control device 12 as the nth frame is the second detected position data, and the speed data sent by the detection data transfer unit 10 to the vehicle control device 12 as the nth frame is the second detected speed data. The vehicle control device 12 controls the vehicle based on the position data, speed data, and object recognition data sent from the detection data transfer unit 10 (step S8).

The object detection device 100A proceeds to generate the next frame for object detection (step S9). As a result, the object detection device 100A repeatedly executes the processes from step S1 to step S9.

In this way, even if the radar detection data of an object using radar 1 is temporarily lost, object detection device 100A generates position data and speed data for the current frame using the radar detection data of the previous frame and the detection data of the current frame using camera 2. In other words, object detection device 100A generates position data and speed data for the current frame based on the position data of the previous frame and the camera speed data of the current frame.

The object detection device 100A can improve the loss of radar detection data by extrapolating the generated position data and speed data as detection data for the current frame. Therefore, the object detection device 100A can improve the object detection performance. The object detection device 100A can detect objects with high accuracy using a camera 2 and an image processor 4A that are small, have low manufacturing costs, and are low load.

When radar detection data is lost, object detection device 100A may generate position data and speed data for the current frame based on position data from two or more previous frames and camera speed data for the current frame. When radar detection data is lost, object detection device 100A generates position data and speed data for the current frame based on position data from the most recent frame possible and camera speed data for the current frame. In this way, when radar detection data is lost, object detection device 100A generates position data and speed data for the current frame based on the most recent stored position data and the camera speed data for the current frame, which is a frame later than the frame of the most recent position data.

Below, we will explain the specific advantages of using the object detection device 100A to detect an object. The object detection device 100A detects the position of an object only using the radar 1 and the signal processor 3, and does not detect the position of an object using the camera 2 and the image processor 4A. This configuration of the object detection device 100A makes it possible to reduce the load on the hardware and software of the camera 2 and the image processor 4A.

To detect and output the position of an object, it is common to measure the distance using a stereo camera, but in object detection device 100A, camera 2 can be configured as a monocular camera. Image processor 4A is configured not to detect position, and processing power can be reduced accordingly. For these reasons, camera 2 and image processor 4A of object detection device 100A can be configured to be compact and have low manufacturing costs because they only require a short processing time to operate.

Here, we will explain the object detection device of the comparative example. Like object detection device 100A, the object detection device of the comparative example does not detect position data using a camera. The object detection device of the comparative example does not have a disappearance determiner 8 and a disappearance extrapolator 9A. Like object detection device 100A, the object detection device of this comparative example also obtains position data using radar, and the position data after fusion processing becomes positive. However, since the object detection device of the comparative example does not have a disappearance determiner 8 and a disappearance extrapolator 9A, if the radar detection data is lost due to a temporary use environment, it cannot detect the position data.

On the other hand, in the object detection device 100A, the disappearance determiner 8 determines whether or not the radar position data and radar speed data acquired using the radar 1 have been lost for the current frame. Also, in the object detection device 100A, the disappearance extrapolator 9A estimates the position data and speed data of the current frame based on the position data stored in the previous frame and the speed data of the camera 2 detected in the current frame, and extrapolates them to the current frame.

As a result, the object detection device 100A can improve the average object detection accuracy even when the camera 2 is configured as a monocular camera that is small and low in manufacturing cost. In other words, the object detection device 100A is capable of high-performance object detection with a small size and low manufacturing cost.

Thus, according to embodiment 1, the object detection device 100A generates current position data and speed data using position data stored in the previous frame and current camera speed data, thereby enabling accurate object detection with a small size and low manufacturing cost.

Embodiment 2.
Next, a second embodiment will be described with reference to Figures 3 and 4. In the second embodiment, only when radar detection data is lost, position data detected by the camera 2 (camera position data, which will be described later) is extrapolated to the current frame.

Figure 3 is a diagram showing the configuration of an object detection device according to embodiment 2. Among the components in Figure 3, components that achieve the same functions as those in the object detection device 100A according to embodiment 1 shown in Figure 1 are given the same reference numerals, and duplicated explanations will be omitted.

Like object detection device 100A, object detection device 100B is a fusion-type object detection device that combines multiple types of sensors, and calculates object detection data based on data obtained from the multiple types of sensors .

The object detection device 100B includes an image processor 4B instead of the image processor 4A. Also, the object detection device 100B includes a fusion processor 5B instead of the fusion processor 5A. That is, the object detection device 100B includes a radar 1, a camera 2, a signal processor 3, an image processor 4B, and a fusion processor 5B.

The fusion processor 5B has an identity determination unit 6B instead of the identity determination unit 6A. In addition, the fusion processor 5B does not have a data storage unit 11A. The identity determination unit 6B has an identity determination unit 7B instead of the identity determination unit 7A, and has an erasure extrapolator 9B instead of the erasure extrapolator 9A.

Below, we will explain the differences between image processor 4B and image processor 4A, and the differences between fusion processor 5B and fusion processor 5A.

Image processor 4B performs object recognition based on image information acquired by camera 2 and calculates the relative speed and position of the object with respect to object detection device 100B. Like image processor 4A, image processor 4B is composed of, for example, an MCU, a CPU, etc. Like image processor 4A, image processor 4B recognizes objects such as people and obstacles using a database of feature data obtained by machine learning or deep learning, and calculates the relative speed and position of the object.

The image processor 4B sends object recognition data indicating that an object has been recognized, camera speed data, and camera position data indicating the position of the object to the fusion processor 5B. Thus, the image processor 4B differs from the image processor 4A in that it calculates the position of the object.

The identity determiner 7B executes an object identity determination process based on the radar position data, radar speed data, camera speed data, camera position data, and object recognition data. If the identity determiner 7B determines that the objects are the same, it associates the radar position data, radar speed data, object recognition data, camera speed data, and camera position data. The identity determiner 7B sends this associated data (hereinafter referred to as second association data) to the disappearance determiner 8 connected in the subsequent stage.

The loss determiner 8 of embodiment 2 determines whether or not there is signal loss in the radar position data and radar speed data, similar to the loss determiner 8 of embodiment 1.

When there is no signal loss, the loss determiner 8 sends the radar position data, the radar speed data, and the object recognition data to the loss extrapolator 9B. When there is a signal loss, the loss determiner 8 sends the object recognition data, the camera speed data, and the camera position data to the loss extrapolator 9B.

The loss extrapolator 9B performs data extrapolation only when the loss determiner 8 detects data loss. When there is no signal loss, the loss extrapolator 9B sends the radar position data, radar speed data, and object recognition data sent from the loss determiner 8 to the detection data transfer unit 10 as the position data, speed data, and object recognition data detected by the object detection device 100B.

When there is a signal loss, the loss extrapolator 9B generates position data and speed data for the current frame, extrapolates the data to the current frame, and transfers the data to the detection data transfer unit 10. Specifically, when there is a signal loss, the loss extrapolator 9B transfers the camera position data, camera speed data, and object recognition data to the detection data transfer unit 10 as the position data, speed data, and object recognition data detected by the object detection device 100B.

Next, the procedure for detecting an object by the object detection device 100B will be described. FIG. 4 is a flowchart showing the procedure for detecting an object by the object detection device according to the second embodiment. Note that, among the processes shown in FIG. 4, the description of the same processes as those described in FIG. 2 will be omitted.

Object detection device 100B executes the processes of steps S1 and S2. In the second embodiment, object detection device 100B executes step S3B instead of step S3A. That is, object detection device 100B recognizes an object using camera 2 and detects the speed and position of the object (step S3B). Specifically, camera 2 captures an image of the object to obtain image information of the object, and outputs the image information to image processor 4B.

Image processor 4B performs object recognition based on image information acquired by camera 2, and generates camera speed data indicating the relative speed of the object with respect to object detection device 100B, and camera position data indicating the relative position of the object with respect to object detection device 100B. Image processor 4B sends object recognition data indicating that the object has been recognized, camera speed data, and camera position data to fusion processor 5B. Note that object detection device 100B simultaneously executes the process of step S2 and the process of step S3B.

The identity determination unit 7B of the fusion processor 5B performs object identity determination processing based on the radar position data, radar speed data, camera speed data, camera position data, and object recognition data (step S4).

If the identity determiner 7B determines that the objects are identical, it generates second association data that associates the radar position data, the radar speed data, the object recognition data acquired using the camera 2, the camera speed data, and the camera position data, and sends the second association data to the disappearance determiner 8.

The disappearance determiner 8 of the object detection device 100B determines whether the radar detection data, which is the radar position data and the radar speed data, has been lost (step S5). If the radar detection data has been lost (step S5, Yes), that is, if there is a signal loss, the disappearance determiner 8 sends the object recognition data, the camera position data, and the camera speed data to the disappearance extrapolator 9B.

When radar detection data is lost, the loss extrapolator 9B extrapolates the camera position data and the camera speed data to the current frame. In other words, when radar detection data is lost, the loss extrapolator 9B adopts the camera position data as the position data of the current frame and adopts the camera speed data as the speed data of the current frame. In this way, the loss extrapolator 9B generates the lost radar detection data and extrapolates it to the current frame (step S6).

After this, the object detection device 100B executes the processes from step S7 onward in the same manner as the object detection device 100A. If the radar detection data has not been lost, the loss extrapolator 9B executes the same processes as the loss extrapolator 9A.

The following describes specific advantages of detecting an object using the object detection device 100B. The object detection device 100B calculates radar position data detected using the radar 1 and camera position data detected using the camera 2.

When camera 2 is a small, low-cost monocular camera, the accuracy of the camera position data and camera speed data detected using camera 2 is generally lower than the accuracy of the radar position data and radar speed data detected using radar 1. For this reason, object detection device 100B uses the detection data of camera 2 only when the detection data of radar 1 is lost. In other words, object detection device 100B extrapolates the camera position data and camera speed data to the current frame only when the radar position data and radar speed data are lost. This allows object detection device 100B to improve the average object detection accuracy even when camera 2 is configured as a small, low-cost monocular camera.

Thus, according to the second embodiment, even if radar detection data is temporarily lost , object detection device 100B estimates radar detection data based on detection data using camera 2 in the current frame. This allows object detection device 100B to improve the loss of radar detection data and improve the average object detection accuracy. Therefore, object detection device 100B can detect objects with high accuracy using camera 2 and image processor 4B that are small, have low manufacturing costs, and are low load.

Here, the hardware configuration of the object detection devices 100A and 100B will be described. The object detection devices 100A and 100B are realized by a processing circuit. The processing circuit may be a processor and memory that executes a program stored in a memory, or may be dedicated hardware such as a dedicated circuit. The processing circuit is also called a control circuit.

Figure 5 is a diagram showing an example of the configuration of a processing circuit when the processing circuit provided in the object detection device according to embodiments 1 and 2 is realized by a processor and a memory. Since object detection devices 100A and 100B have similar hardware configurations, the hardware configuration of object detection device 100A will be described below.

The processing circuit 90 shown in FIG. 5 is a control circuit and includes a processor 91 and a memory 92. When the processing circuit 90 is configured with the processor 91 and the memory 92, each function of the processing circuit 90 is realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 92. In the processing circuit 90, the processor 91 reads and executes the program stored in the memory 92 to realize each function. That is, the processing circuit 90 includes a memory 92 for storing a program that results in the processing of the object detection device 100A being executed. This program can also be said to be a program for causing the object detection device 100A to execute each function realized by the processing circuit 90. This program may be provided by a storage medium in which the program is stored, or may be provided by other means such as a communication medium. The above program can also be said to be a program for causing the object detection device 100A to execute the object detection process.

The processor 91 is, for example, a CPU (also called a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, or DSP (Digital Signal Processor)) or a system LSI (Large Scale Integration).

The memory 92 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), an EEPROM (registered trademark) (Electrically EPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

Figure 6 is a diagram showing an example of a processing circuit when the processing circuit provided in the object detection device according to the first and second embodiments is configured with dedicated hardware. The processing circuit 93 shown in Figure 6 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these. The processing circuit 93 may be partially realized with dedicated hardware and partially realized with software or firmware. In this way, the processing circuit 93 can realize each of the above-mentioned functions by dedicated hardware, software, firmware, or a combination of these. Also, as described above, the signal processor 3 and the image processor 4A may each be configured with a separate CPU or the like.

The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or the embodiments may be combined with each other. Also, parts of the configurations may be omitted or modified without departing from the spirit of the invention.

1 radar, 2 camera, 3 signal processor, 4A, 4B image processor, 5A, 5B fusion processor, 6A, 6B identity determination unit, 7A, 7B identity determiner, 8 disappearance determiner, 9A, 9B disappearance extrapolator, 10 detection data transfer unit, 11A data storage unit, 12 vehicle control device, 90, 93 processing circuit, 91 processor, 92 memory, 100A, 100B object detection device.

Claims (3)

A radar that emits electromagnetic waves to an object and receives a reflected signal from the object;
a signal processor that calculates radar position data indicating a position of the object and radar speed data indicating a speed of the object based on the reflected signal;
a camera that captures an image of the object to obtain image information of the object;
an image processor that calculates camera speed data indicative of a speed of the object based on the image information;
a fusion processor that outputs the radar position data and the radar speed data in a first frame to an external device as first detected position data indicating a position of the object in the first frame and first detected speed data indicating a speed of the object in the first frame;
Equipped with
the fusion processor has a data storage device for storing the first detected position data;
when the radar position data and the radar speed data are lost in a second frame that is a frame subsequent to the first frame, generate second detected position data indicating the position of the object in the second frame and second detected speed data indicating the speed of the object in the second frame based on the first detected position data and the camera speed data acquired in the second frame, and output the data to the external device;
1. An object detection device comprising: The image processor calculates orientation data indicating an orientation of the object based on the image information,
the fusion processor has an identity determiner that performs identity determination processing to determine whether the objects detected by the radar and the camera are the same object;
the identity determiner executes identity determination processing of the object based on the radar position data, the radar speed data, the camera speed data, and the orientation data;
2. The object detection device according to claim 1 . a signal processing step in which an object detection device including a radar that emits electromagnetic waves to an object and receives a reflected signal from the object, and a camera that captures an image of the object to obtain image information of the object, calculates radar position data indicating the position of the object and radar speed data indicating the speed of the object based on the reflected signal;
an image processing step in which the object detection device calculates camera speed data indicating a speed of the object based on the image information;
a fusion processing step in which the object detection device outputs the radar position data and the radar speed data in a first frame to an external device as first detected position data indicating the position of the object in the first frame and first detected speed data indicating the speed of the object in the first frame;
Including,
In the fusion processing step, the object detection device stores the first detected position data, and when the radar position data and the radar speed data are lost in a second frame that is a frame subsequent to the first frame, generates second detected position data indicating the position of the object in the second frame and second detected speed data indicating the speed of the object in the second frame based on the first detected position data and the camera speed data acquired in the second frame, and outputs the data to the external device.
13. An object detection method comprising:

Images