JP7789231B2 - In-vehicle information processing device, in-vehicle information processing system, and in-vehicle information processing method - Google Patents

Description

The present invention relates to an in-vehicle information processing device, an in-vehicle information processing system, and an in-vehicle information processing method.

In recent years, the development of technologies necessary for vehicle driving assistance and autonomous driving has been progressing, and one of the elemental technologies for these technologies is the technology to grasp the vehicle's surroundings. As driving assistance and autonomous driving become more advanced, it has become important to grasp the vehicle's surroundings in more detail, and as a means to achieve this, a large number of external sensors such as cameras, millimeter-wave radar, and LiDAR have come to be used.

In driving assistance and autonomous driving, external sensing data acquired by these external sensors is subjected to recognition processing to detect obstacles to be avoided and areas in which the vehicle should be driven, and the recognition results are then comprehensively judged, taking into account the vehicle driver's operations as necessary, to ultimately determine the necessary action.

Patent document 1 describes a technology that acquires parallax images using a stereo camera, derives the coordinates of an object in a world coordinate system from the parallax images, and determines whether an object located in the vehicle's planned travel area is a person by using the object as a priority object.

JP 2020-160914 A

As the number of external sensors increases, the amount of external sensing data to be processed for recognition also increases, increasing the amount of calculation required for recognition processing, and reducing the computational load becomes an issue. In particular, when handling high-resolution data or performing high-precision recognition processing, the amount of calculation required for recognition processing increases even for each piece of sensing data from an individual external sensor, creating a growing demand for reducing the computational load.
One method for reducing the amount of calculation is to perform recognition processing by prioritizing areas sensed by external sensors, thereby improving the efficiency of the recognition processing. A technique described in Patent Literature 1 is known as such a method.
However, when providing driving assistance or controlling autonomous driving for vehicles traveling on public roads, there are situations where there are many objects in the vicinity, and in such situations, there are many priority objects, making it difficult to reduce the amount of calculation.
It is also necessary to respond to fast-moving objects in the vicinity, such as bicycles and cars, and in cases where a surrounding object approaches, recognition of the priority object may be delayed, leading to delays in control.
The object of the present invention is to realize an in-vehicle information processing device, an in-vehicle information processing system, and an in-vehicle information processing method that can reduce recognition delays for objects with high urgency by increasing the amount of computing resources for areas with high priority compared to areas with low priority.

In order to achieve the above object, the present invention is configured as follows.

The in-vehicle information processing device includes a control unit, and the control unit includes an external world data acquisition unit that acquires external world data from external world sensors mounted on the vehicle, a point cloud cluster generation unit that performs clustering on a plurality of points constituting the external world data acquired from all or some of the external world sensors to generate a point cloud cluster, a priority setting unit that sets a priority to the point cloud cluster or an external world area where the point cloud cluster exists based on the relative speed of the point cloud cluster with respect to the vehicle calculated from the external world data related to the point cloud cluster, a calculation resource amount determination unit that determines the amount of calculation resources for recognition processing of the external world data in accordance with the priority, and a calculation resource amount determination unit that calculates an image of the external world data acquired by the external world data acquisition unit. an image recognition processing unit that recognizes the image corrected by the image correction processing unit based on the amount of computational resources determined by the computational resource amount determination unit; a point cloud cluster recognition processing unit that performs object type detection and noise removal on the point cloud cluster generated by the point cloud cluster generation unit and performs recognition processing on the point cloud cluster; and a recognition result integration processing unit that checks the consistency of the recognition results between the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit, selects and removes inconsistent recognition results, and integrates the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit to output the result as an external world recognition result .

The in-vehicle information processing system also includes a plurality of external sensors that detect objects around the vehicle, and a control unit having a configuration similar to that of the control unit of the in-vehicle information device.

and a control unit of the on-board information processing method for detecting objects around a vehicle using a plurality of external sensors. The control unit of the on-board information processing device performs clustering on a plurality of points constituting the external data acquired from all or some of the external sensors to generate a point cloud cluster. The control unit of the on-board information processing device performs clustering on the plurality of points constituting the external data acquired from all or some of the external sensors to generate a point cloud cluster. The control unit of the on-board information processing device performs clustering on the plurality of points constituting the external data acquired from all or some of the external sensors to generate a point cloud cluster. The control unit of the on-board information processing device performs clustering on the plurality of points constituting the external data acquired from all or some of the external sensors to generate a point cloud cluster. The control unit of the on-board information processing device performs clustering on the plurality of points constituting the external data acquired from all or some of the external sensors to generate a point cloud cluster. The control unit of the on-board information processing device performs clustering on the plurality of points constituting the external data acquired from all or some of the external sensors to generate a point cloud cluster. The control unit of the on-board information processing device performs clustering on the plurality of points constituting the external data acquired from all or some of the external sensors to generate a point cloud cluster. The control unit of the on-board information processing device performs clustering on the plurality of points constituting the external world cluster

It is possible to realize an in-vehicle information processing device, an in-vehicle information processing system, and an in-vehicle information processing method that can appropriately determine priority according to the surrounding area being sensed, even in a vehicle traveling on a public road, and allocate more computing resources to areas with higher priority than to areas with lower priority, thereby suppressing delays in recognition of objects with high urgency. Furthermore, by using computing resources to improve recognition accuracy and recognition function, it is possible to suppress the occurrence of erroneous recognition or non-recognition of objects with high urgency, and to obtain detailed information such as the type and behavior of the object.

1 is a diagram showing an example of a sensor configuration of an external sensing system to which the present invention is applied; 1A to 1C are diagrams illustrating an example of the operation of the external sensing system according to the present invention. FIG. 1 is a diagram illustrating an example of a connection configuration of an external sensing system according to a first embodiment. FIG. 2 is a functional block diagram of a control unit according to the first embodiment. FIG. 2 is a diagram showing a processing flow of the external sensing system according to the first embodiment. FIG. 10 is a diagram illustrating an example of region division of an image captured by a camera. FIG. 10 is a diagram illustrating an example of the order of recognition processing of a camera-acquired image. FIG. 10 is a diagram illustrating an example of a sensor configuration of an external sensing system according to a second embodiment. FIG. 10 is a diagram illustrating an example of priority weighting of recognition processing depending on the state of a vehicle. FIG. 10 is a diagram illustrating an example of priority weighting of recognition processing depending on the state of a vehicle. FIG. 10 is a diagram illustrating an example of priority weighting of recognition processing depending on the state of a vehicle. FIG. 10 is a diagram illustrating an example of priority weighting of recognition processing depending on the state of a vehicle. FIG. 10 is a diagram illustrating an example of priority weighting of recognition processing depending on the state of a vehicle. FIG. 10 is a diagram illustrating an example of a sensor configuration of an external sensing system according to a third embodiment. FIG. 1 is a diagram illustrating an example of a processing flow of an external sensing system using multiple ECUs. FIG. 10 is a diagram illustrating an example of image data transfer settings via an inter-ECU image transfer compatible communication path. FIG. 10 is a diagram illustrating a usage setting of a communication path compatible with image transfer between ECUs. FIG. 10 is a diagram illustrating an example of a sensor configuration of an external sensing system according to a fourth embodiment. FIG. 11 is a diagram illustrating an example of dividing the angle of view of a camera in a connection configuration example according to a third embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Example 1
An example of the sensor configuration of an external sensing system to which the present invention is applied will be described with reference to FIG.

1, the vehicle 10 has six cameras and a LiDAR A290 capable of sensing the entire periphery as external sensors that detect objects around the vehicle 10. The six cameras are a camera F310 for monitoring the front, a camera FL320 for monitoring the left front, a camera FR330 for monitoring the right front, a camera RL340 for monitoring the left rear, a camera RR350 for monitoring the right rear, and a camera R360 for monitoring the rear.

The horizontal angle of view of camera F310 is shown as camera angle of view F410, the horizontal angle of view of camera FL320 as camera angle of view FL420, the horizontal angle of view of camera FR330 as camera angle of view FR430, the horizontal angle of view of camera RL340 as camera angle of view RL440, the angle of view of camera RR350 as camera angle of view RR450, and the horizontal angle of view of camera R360 as camera angle of view R460.

An example of the operation of the external sensing system according to the present invention will be described using Fig. 2. In Fig. 2, there are three surrounding vehicles: vehicle A20, vehicle B30, and vehicle C40. Vehicle A is approaching the host vehicle 10 at a speed Va, vehicle B is moving away at a speed Vb, and vehicle C is approaching at a speed Vc. In this situation, the host vehicle 10 uses LiDAR A290 to acquire a point cloud including distance information reflecting the surrounding vehicles.

Then, a point cloud cluster, which is a collection of spatially nearby points, is created, and a representative relative distance from the vehicle 10 is obtained for each point cloud cluster. Furthermore, a change in the relative distance over time is obtained for each point cloud cluster, and the relative speed with respect to the vehicle 10 is calculated.

If the LiDAR A290 can directly measure the relative velocity by sensing, the relative velocity obtained by sensing may be used instead of the relative velocity calculated from the change in relative distance over time. Also, any external sensor other than LiDAR may be used as long as it can obtain the information necessary to create a point cloud cluster.

A point cloud cluster may be a collection of pixels, which are the points that make up an image. In other words, a point cloud cluster may be created based on image data acquired by a camera. However, since distance information is required to determine whether pixels are spatially close, distance information must be provided to each pixel using techniques such as stereo vision using multiple cameras before use. In this case, pixels for which distance information cannot be obtained using stereo vision or other methods must be excluded or estimated from distance information of neighboring pixels.

Based on the direction and relative speed at which the object was detected by LiDAR A290, the recognition process prioritizes image data acquired by cameras F310, FL320, and R360, which correspond to camera view angles F410, FL420, and R460 that overlap greatly with the direction in which the object approaching the vehicle 10 is located.

Judging from the relative speed, the priority of the recognition process is not increased for image data acquired by camera FR330, which is a camera corresponding to camera angle of view F430 that has a large overlap with only the direction of objects moving away from the host vehicle 10. By increasing the priority of the recognition process only for images corresponding to vehicles A20 and C40 without increasing the priority of the recognition process for images that only contain objects moving away from the host vehicle 10, it becomes possible to allocate more computing resources to the recognition process for vehicles A20 and C40, thereby improving the recognition accuracy for objects that may have an impact on the host vehicle 10 and shortening the time required for recognition.

The priority of the recognition process may be increased for smaller units than the image data units acquired by each camera. For example, the unit may be a unit obtained by dividing the image data acquired by each camera into multiple units, or it may be only image data in an area on the image corresponding to the part where the target point cloud cluster exists. When increasing the priority of the area on the image corresponding to the part where the point cloud cluster exists, it may be possible to set the area of sensing data where the recognition process is performed slightly larger than the area where the target point cloud cluster exists, taking into account the positional deviation between the external sensor that obtains the sensing data used to create the point cloud cluster and the external sensor that obtains the sensing data used to perform the recognition process.

By using the relative speed (positive for the direction away from the vehicle 10 and negative for the direction towards it) in combination with the relative distance, for example, by dividing the relative distance by the relative speed and inverting the sign, the time until collision (hereinafter referred to as the collision grace period) assuming that the state at the time of sensing continues can be calculated, and it is also possible to set the priority of the recognition process in stages.

For example, it is considered that the smaller the positive value of the collision grace period for a point cloud cluster, the shorter the time grace period in case of a collision. Therefore, an object corresponding to such a point cloud cluster can be determined to be a recognition target with a high degree of urgency, and a higher priority can be given to the recognition process for image data acquired by a camera corresponding to an angle of view that covers the direction in which the point cloud cluster exists.

If the relative velocity of a point cloud cluster is positive, i.e., if the collision grace period is negative, it indicates that the object corresponding to the point cloud cluster is moving away, so the recognition processing priority is not increased for image data acquired by a camera corresponding to an angle of view that covers the direction in which the point cloud cluster exists.

When there are multiple point cloud clusters in the corresponding angle of view of a certain camera, the priority of the recognition process for the image acquired by that camera is set to the point cloud cluster with the smallest positive collision grace period among those multiple point cloud clusters, i.e., the point cloud cluster with the shortest time grace period in the event of a collision.

When the relative distance is divided by the relative speed to obtain the collision grace period, if the relative speed is close to 0, the collision grace period becomes very large, and if the relative speed is 0, the collision grace period becomes infinite. In such a case, clipping is performed to obtain a large fixed value that will not be treated as a target for increasing the priority in the priority adjustment of the recognition processing, thereby preventing the collision grace period from becoming infinite.

In the situation shown in FIG. 2, if the priority of the recognition process is determined using the collision margin time, the result will be as follows, for example.

Since vehicle B30 is moving away from the host vehicle 10, although it is the closest to the host vehicle 10 among the surrounding vehicles, this does not become a factor for increasing the priority of the recognition process. Between vehicle A20 and vehicle C40, vehicle C40 is closer to the host vehicle 10, but the relative speed Va of vehicle A20 is greater than the relative speed Vc of vehicle C40. In the situation shown in FIG. 2 , a relationship between the relative speeds Va and Vc is assumed such that the collision grace period of vehicle A20 is shorter than the collision grace period of vehicle C40. In this situation, it is determined that the priority of the recognition process is high for the direction in which there is a point cloud cluster corresponding to vehicle A20, which is farther away than vehicle C40 but has a shorter collision grace period.

As a result, the recognition process priority is given highest to image data acquired at camera view angles F410 and FL420, which overlap closely with the orientation of vehicle A20, and the recognition process priority is given next to image data acquired by camera R360, which corresponds to camera view angle R460, which overlaps closely with the orientation of vehicle C40. In other words, compared to when the recognition process priority is simply determined based on the relative distance to the object, it is possible to determine the recognition process priority in an appropriate manner corresponding to the level of urgency.

Although a simple method using relative distance and relative speed has been shown for calculating the time to wait for collision, it is also possible to estimate the time to wait for collision using more advanced methods, such as taking into account changes over time in the relative speed and relative direction with respect to the object, and in such cases it is also possible to lower the level of urgency for objects that have become less likely to collide.

Fig. 3 shows an example of a connection configuration of an external world sensing system that realizes the example operation described using Fig. 2. As shown in Fig. 3, external world sensors LiDAR A290, camera F310, camera FL320, camera FR330, camera RL340, camera RR350, and camera R360 are directly connected to ECU Type A110 (control unit) that has a signal processing function including external world recognition processing.

It is necessary to transmit point cloud data from the LiDAR A290 and image data from the cameras F310, FL320, FR330, RL340, RR350, and R360 to the ECU Type A110 at a frame rate of approximately several tens of frames per second. For this reason, a direct connection is made to ensure the necessary sensing data communication bandwidth and to suppress communication delays. High-speed serial communication of several to several tens of Gbps is used for communication between the external sensors and the ECU Type A110.

The ECU Type A110 also communicates with other ECUs installed in the vehicle 10 via an interface such as a CAN. By connecting with other ECUs, the ECU Type A110 can transmit the results of recognizing the outside world to ECUs that make driving decisions and control driving, and can acquire the driving status of the vehicle. Depending on the processing capacity of the ECU, the ECU Type A110 may also perform processes other than those related to external world sensing.

FIG. 4 is a schematic internal functional block diagram of the ECU Type A110.

4, the ECU Type A110 includes an external data acquisition unit 111 that acquires external data from external sensors 290, 310 to 360 mounted on the vehicle 10, a point cloud cluster generation unit 112 that performs clustering on a plurality of points constituting the external data for which three-dimensional position information is available among the external data acquired by the external sensor acquisition unit 111 from the external sensors 290, 310 to 360, and generates a point cloud cluster, an image correction processing unit 116 that performs corrections such as image distortion of the external data, image brightness correction, color reproduction and correction, and image misalignment, and a priority setting unit 113 that sets a priority to the point cloud cluster or an external area in which the point cloud cluster is located, based on the relative speed of the point cloud cluster with respect to the vehicle 10 obtained from the external data related to the point cloud cluster generated by the point cloud cluster generation unit 112.
Furthermore, the ECU Type A110 includes a computational resource amount determination unit 114 that determines the amount of computational resources for recognition processing of external world data in accordance with the priority set by the priority setting unit 113, an image recognition processing unit 117 that performs recognition processing on images, a point cloud cluster recognition processing unit 118 that performs recognition processing on point cloud clusters generated by the point cloud cluster generation unit 112 based on point cloud data acquired from the LiDAR A290, and a recognition result integration processing unit 115 that integrates the processing results of the point cloud cluster recognition processing unit 118 and the processing results of the image recognition processing unit 117 and outputs the result as an external world recognition result.

An example of processing performed by ECU Type A110 is shown in Figure 5. Figure 5 is divided into LiDAR point cloud processing and camera image processing, and each shows the processing for one frame. In other words, the processing shown in Figure 5 is repeated for each frame. LiDAR point cloud frame processing processes external sensing data acquired by LiDAR A290, and camera image frame processing processes image data acquired by camera F310, camera FL320, camera FR330, camera RL340, camera RR350, and camera R360.

The recognition results performed by LiDAR point cloud processing and camera image processing are sent to the recognition result integration processing unit 115, where fusion processing is performed to obtain the final recognition results of the surrounding conditions of the vehicle 10.

When LiDAR point cloud processing, camera image processing, and recognition result integration processing are executed within the ECU, they may be executed as distributed processing using multiple processor cores, or as time-sharing processing, or as a combination of distributed processing and time-sharing processing.

Although time-sharing or distributed processing may be used to process the image data obtained from multiple cameras, the amount of computing resources allocated to the recognition processing of each image, i.e., the number of available processor cores, the available processing time, and the allocation of various dedicated processing circuits, is adjusted based on the recognition priority.

In the LiDAR point cloud frame processing, external sensing data acquired by the LiDAR A290 in a LiDAR point cloud data acquisition step S710 is imported into the external data acquisition unit 111, and in a point cloud acquisition signal processing step S720, the point cloud cluster generation unit 112 performs error correction on the imported data due to individual differences between sensors, errors in mounting the sensor on the vehicle, etc. Also, noise contained in the sensing signal is removed, such as by filtering input signals that do not meet a certain level.

Next, in point cloud clustering processing step S730, the point cloud cluster generation unit 112 clusters the point cloud. Point cloud clustering is performed by, for example, detecting surrounding points within a certain distance in space from a given point, and if there are a certain number or more of such surrounding points, designating the point as a valid point and then repeatedly classifying the point as belonging to the same cluster as the detected surrounding points. However, since it is necessary to separate objects from the road surface, the road surface is estimated in advance taking into account the relationship between the sensor position and the horizontal plane, and points located below the road surface in space are excluded. Furthermore, if the number of points constituting a cluster as a result of clustering is below a certain number, the point cloud cluster is excluded as it is considered to be noise.

In the point cloud clustering process S730, for each extracted point cloud cluster, several to several tens of points constituting the cluster are selected in order of shortest relative distance from the host vehicle 10, and the relative distances of these points are averaged to determine the relative distance from the host vehicle 10 to the point cloud cluster. Since contact of even a part of an object corresponding to a point cloud cluster with the host vehicle 10 indicates a collision, the value of the smallest relative distance among the points constituting the point cloud cluster is used, but since the influence of noise is large at a single point, the relative distance to the point cloud cluster is determined using the average of multiple points.

The point cloud clusters extracted in the point cloud clustering processing step S730 are associated with the point cloud clusters of the previous frame in the point cloud cluster relative velocity detection step S740, and the relative velocity is calculated from the difference between the elapsed time and relative distance from the previous frame. The relative velocity is considered to be a change in relative distance over time, and the approach direction is set to a negative value. The association with the point cloud clusters of the previous frame can be performed, for example, by associating the point cloud cluster with the most overlapping points, or by estimating the position of the point cloud cluster of the previous frame from the elapsed time taking into account the movement tendency of the point cloud cluster up to the previous frame, and then associating the point cloud cluster with the most overlap with the estimated position.

In the point cloud cluster relative velocity detection step S740, if millimeter wave radar is used instead of LiDAR, or if the sensing method is capable of obtaining relative velocity even with LiDAR, the relative velocity obtained as a sensing result may be used.

After obtaining the relative speed of the peripheral object with respect to the host vehicle 10 in the point cloud cluster relative speed detection step S740, the priority setting unit 113 calculates the collision grace time for each point cloud cluster by dividing the relative distance by the relative speed and inverting the sign for each point cloud cluster in the collision grace time calculation step S750. That is, the priority setting unit 113 uses the relative speed to calculate the collision grace time that indicates the time required for the point cloud cluster to come into contact with the host vehicle 10, and sets a priority for the point cloud cluster or the external area in the direction in which the point cloud cluster is located based on the collision grace time.

In addition, in preparation for the case where the absolute value of the relative velocity is very small and the collision grace period is very large, for example, if the collision grace period exceeds 30 seconds, processing such as clipping at 30 seconds is performed. Also, even if the calculated collision grace period is negative, that is, if the relative velocity is positive, the collision grace period is set to the maximum possible value of 30 seconds, and the point cloud cluster corresponding to the moving away object is treated as a point cloud cluster with a long collision grace period, thereby preventing the recognition process from having a high priority.

In the collision grace period notification step S760, the range of the direction in which each point cloud cluster exists is calculated for each point cloud cluster, and the calculated range of the direction is notified to the recognition processing priority setting step S735 together with the collision grace period of the point cloud cluster, and the priority is set by the priority setting unit 113. The notification is performed by a method suitable for the implementation form, such as a variable, a shared memory area, or inter-process communication.

Regarding the orientation, in the first embodiment, image data is acquired by multiple cameras, with the horizontal direction being divided, so when performing recognition processing in units of image data acquired by each camera, only the horizontal direction is required. However, when a configuration using a separate camera in the vertical direction is included, the vertical direction orientation is also required. Furthermore, when images captured by individual cameras are divided horizontally and recognition processing is performed separately for the top and bottom, the vertical direction orientation is also required to determine the priority of recognition processing for each divided image region. When determining the priority of recognition processing for each point cloud cluster, horizontal and vertical direction information is also required to identify the range in which the recognition processing target exists.

In the point cloud cluster recognition processing step S770, the point cloud cluster recognition processing unit 118 performs a recognition process for each point cloud cluster by referring to the point cloud data constituting the point cloud cluster. Based on the spatial positional relationship of each point data constituting the point cloud cluster and the distribution of the signal intensity of the reflected laser light, the type of object is detected and point cloud clusters that appear to be noise are excluded.

The recognition priority may also be taken into consideration when performing the point cloud cluster recognition process. That is, the recognition process may be performed in order of the point cloud clusters with the shortest collision time allowed in the collision time calculation step S750, or a relatively light recognition process may be performed on point cloud clusters with long collision time allowed, in which only noise removal processing is performed to detect the presence of objects, and a relatively heavy recognition process may be performed on point cloud clusters with collision time allowed less than a certain time to detect the type of vehicle (four-wheeled or larger vehicle, two-wheeled vehicle, bicycle, pedestrian, etc.).

By performing the recognition process in order from the point cloud cluster with the shortest collision grace period, it is possible to obtain the advantage of quickly obtaining the recognition result for the object with the highest degree of urgency. In addition, by performing the recognition process to identify the object type after the collision grace period is equal to or less than a certain time, it is possible to perform the high-load recognition process only for the objects near the vehicle 10, which have a large impact on the travel route determination and the collision avoidance operation, and it is possible to reduce the overall recognition process load.

The recognition results of the point cloud cluster recognition processing step S770 are sent to a recognition result integration process that integrates the recognition results for the sensing data obtained from multiple external sensors, and the final external recognition result is obtained after synthesizing the recognition results for various directions of the vehicle 10, checking the consistency of the recognition results from the multiple external sensors, and selecting or excluding recognition results where inconsistencies occur.

In the camera image processing, first, in a camera image data acquisition step S715, the external world data acquisition unit 111 acquires image data from each camera. In an image correction processing step S725, the image correction processing unit 116 corrects the image data acquired from each camera for image distortion, image brightness correction, color reproduction and correction, image misalignment, etc. At this time, corrections due to individual differences between each camera and errors in installing the camera on the vehicle are also performed. Regarding color, when using image data acquired by an image sensor that uses color filters with different transmission wavelength characteristics such as red, green, and blue that vary depending on the pixel, color restoration is performed using multiple surrounding pixels.

In the image correction processing step S725, image data acquired from a certain camera may be divided into multiple image regions and processed as necessary. Also, different region divisions may be performed for each camera. For example, if a certain camera acquires an image using a fisheye lens with a wide angle of view, the distortion of the image data acquired by that camera will differ significantly between the center and periphery of the image. Therefore, it is conceivable to divide the image data for that camera into a center region and multiple periphery regions, cut out each region as a separate image, and perform appropriate distortion correction for each region.

In the recognition processing priority setting step S735, the priority setting unit 113 determines the priority of the recognition processing for each piece of image data output from the image correction processing step S725 based on the range of directions of each point cloud cluster and the collision allowance time obtained in the collision allowance time notification step S760. However, each piece of image data output from the image correction processing step S725 may be divided into multiple regions on the image, and the priority may be determined for each divided unit.

The recognition priority is determined by the unit of image data to be processed for recognition, and the point cloud cluster that overlaps with the direction included in the image in a certain direction or more is set as the point cloud cluster to be considered, and the collision grace period of the point cloud cluster is used to determine the priority. The unit of image data to be processed for recognition may be a range corresponding to a point cloud cluster, in which case the priority is determined by the collision grace period of the point cloud cluster.

The criteria for the degree of overlap when determining whether a point cloud cluster should be considered are determined, for example, taking into account the specifications of the recognition process. That is, the minimum proportion of the object contained in the image required for object recognition in the recognition process is taken into account. If the object portion is not included in a proportion that is recognizable in the recognition process, the object cannot be recognized in the recognition process, so information about point cloud clusters corresponding to such objects is excluded and handled.

A simple method for determining the priority of recognition is to treat a collision time that is less than a certain value as a high priority. When dividing the priority of recognition into multiple stages, the collision time value is divided into stages and a tiered priority is assigned to each level. For example, it is conceivable to divide the priority into five stages such as urgent, highest, high, medium, and low, divide the collision time range into ranges, and assign the priority in order of shortest time.

In the recognition processing priority setting step S735, when determining the priorities, the priorities may be weighted based on the driving state and operation state of the vehicle 10. For example, during normal driving, it is conceivable to give a higher priority to images acquired by camera F310 corresponding to the front, and during reversing, give a higher priority to images acquired by cameras R360, RL340, and RR350 corresponding to the rear and diagonally rear. Because the vehicle 10 often makes large turns when reversing, such as when parking in a garage, the priority of diagonally rearward images is also increased. Because the angles of view of camera RL340 and camera RR350 include some of the lateral and forward views, the images acquired by these cameras may be vertically divided, and only the images in the area corresponding to the rearward view may be given a higher priority.

When performing recognition processing on image data in units of point cloud clusters, it is also possible to perform recognition processing preferentially on the area corresponding to the point cloud cluster on the image acquired by the camera with the highest priority weighting among cameras that take images including the orientation in which the point cloud cluster exists.

In the recognition computation resource amount determination step S745, the computation resource amount determination unit 114 determines the computation resources to be allocated to the recognition processing for each image data unit of the recognition processing performed in the image recognition processing step S755 based on the recognition processing priority determined in the recognition processing priority setting step S735. The computation resources include the processing time, the number of processors used for the processing, the dedicated circuits used for the recognition processing, the memory access bandwidth and memory capacity used for the operation of the processors and dedicated circuits, etc. The order in which the recognition processing is performed is also determined. The computation resource amount determination unit 114 determines the computation resource amount for the recognition processing for each of the multiple image regions (or each image) captured by the cameras F310, FL320, FR330, RL340, RR350, and R360 according to the priority.

In order to allocate the amount of computational resources according to priority, for low-priority recognition processes, the resolution of the image data used in the recognition process may be lowered, or the frequency of actual recognition processing may be reduced by thinning out frames to be subjected to recognition processing (omitting recognition processing for frames at the timing of thinning out), or recognition processes with logics that differ in recognition performance (recognition rate) and recognition functionality (number of types of objects that can be distinguished) may be prepared, and logic with lower performance and functionality but lighter computational processing may be used. It is possible to accommodate multiple priorities by varying the degree to which the resolution is lowered, the frequency with which frames are thinned out, and switching to recognition logic with different computational processing loads.

When the recognition frequency is reduced, in order to prevent the recognition process from concentrating on processing of a specific frame or from not performing recognition processing of images in a specific direction relative to the vehicle 10 for a long period of time, the process is carried out by taking into consideration multiple frames and planning when and in which direction to perform recognition processing of image data.

The order of recognition processing is normally such that the highest priority is performed first, but if it is detected that recognition processing of images in a specific direction has been continuously thinned out for a long period of time, processing of image data in that direction will be performed first except for image data with a particularly high priority, such as an urgent priority. By performing recognition processing in this manner, it is possible to avoid a situation where recognition processing of image data in a certain direction is not performed for a long period of time, even if there is an error in allocating the amount of computing resources and the recognition processing is terminated midway due to a lack of processing time.

In image recognition processing step S755, the image recognition processing unit 117 performs recognition processing on each image data based on the content determined in recognition computation resource amount determination step S745. Recognition processing is omitted for image data for which it was determined in recognition computation resource amount determination step S745 that recognition processing should be omitted. When performing recognition processing on multiple image data, each image data may be processed sequentially, or parallel processing may be performed using multiple processor cores. Also, parallel processing and sequential processing may be combined, such as parallel processing for two sets of image data at a time and processing these parallel processes sequentially. If the recognition processing time becomes longer than expected and it is likely that processing of the next frame will be affected, the processing is aborted.

The recognition result obtained in the image recognition processing step S755 is sent to a recognition result integration processing unit 115 that integrates the recognition results for the sensing data obtained from the multiple external sensors 290, 310 to 360, and after checking the consistency of the recognition results and selecting or excluding inconsistent recognition results, the final external recognition result is obtained.

6 and 7 show an example in which image data acquired by camera F310, camera FL320, and camera FR330 is divided into multiple image regions and handled, and recognition processing with different priorities is performed for each region by changing the recognition frequency.

6, an image acquired by camera FL320 is divided into areas D514 and F516 for handling. An image acquired by camera F310 is divided into areas A511, B512, and C513 for handling. An image acquired by camera FR330 is assumed to be divided into areas E515 and G517 for handling.

Fig. 7 shows an example of the order in which recognition processing is performed for each area shown in Fig. 6. The letters A to G in Fig. 7 correspond to areas A511 to G517 shown in Fig. 6, respectively, and indicate that recognition processing is performed on the unit enclosed in square brackets. For example, [B, A, C] 612 indicates that recognition processing is performed on area B512, area A511, and area C513 collectively. In other words, this indicates that recognition processing is performed on the entire image acquired by camera F310. [A] 611 indicates that recognition processing is performed on only area A511.

By grouping the areas of images acquired by the same camera together as enclosed in square brackets and performing recognition processing all at once, even if the image of a certain object is only partially contained in each image area due to image area division, it is possible to perform recognition processing without being affected by area division when processing the images together, thereby improving the recognition rate for images.

In Figure 7, it is assumed that the vehicle 10 is moving straight ahead at high speed, and the order in which the recognition process is performed is such that area A511 has high priority, area B512, area C513, area D514, and area E515 have medium priority, and area F516 and area G517 have low priority.

According to this order, in the eight consecutive recognition processes shown in FIG. 7, the recognition process can be performed four times on area A511, two times on area B512, area C513, area D514, and area E515, and one time on area F516 and area G517.

As shown in Figure 7, by appropriately adjusting the order of recognition processes according to the priority of the recognition processes, it is possible to increase the recognition frequency of high-priority areas while suppressing the influence of area division of images acquired with the same camera.

In the first embodiment, the in-vehicle information processing device includes an ECU Type A110 (controller).

According to the first embodiment of the present invention, it is possible to realize an in-vehicle information processing device, an in-vehicle information processing system, and an in-vehicle information processing method that can reduce recognition delays for objects with high urgency by increasing the amount of computational resources for areas with high priority compared to areas with low priority.

In addition, in Example 1, the adjustment of the priority for recognizing images acquired by a camera was shown, but even when performing recognition processing on sensing data acquired by other external sensors such as LiDAR or millimeter wave radar to detect object types, etc., the generation of a point cloud cluster is performed before the recognition processing, so it is possible to adjust the priority of the recognition processing using the relative speed with respect to the point cloud cluster and the collision grace time.

Example 2
Next, a second embodiment of the present invention will be described.

A second embodiment will be described in which the LiDARs are changed from the configuration using the LiDAR A290 shown in the first embodiment to a LiDAR F210 for forward monitoring and a LiDAR R260 for rearward monitoring.

FIG. 8 is a diagram illustrating an example of a system connection configuration according to the second embodiment.

In Example 2, the vehicle 10 has six cameras as external sensors, a LiDAR F210 for forward monitoring, and a LiDAR R260 for rearward monitoring. The angle of view of each camera is the same as in Example 1. Unlike in Example 1, the LiDAR is for forward monitoring and rearward monitoring, and there is no LiDAR for left and right monitoring. Due to the change in the external sensor configuration, the ECU having a signal processing function including external recognition processing is changed from ECU Type A110 to ECU Type B120 (control unit).

The ECU Type B 120 also communicates with other ECUs mounted on the vehicle 10 via an interface such as a CAN. By connecting with other ECUs, it is possible to transmit the results of recognizing the outside world to ECUs that make driving decisions and control driving, and to obtain the driving status of the vehicle 10.

In Example 2, the method for determining the priority of the recognition process in the range that can be sensed by the LiDAR F210 and the LiDAR R260 is the same as the method shown in Example 1. That is, point cloud clusters are generated from point cloud data obtained by the LiDAR F210 or the LiDAR R260, a collision grace period is calculated for each point cloud cluster, and the priority of the recognition process is increased for camera images corresponding to the direction of a point cloud cluster that shows an object with a short collision grace period.

The left and right sides that are not covered by LiDAR are addressed by weighting the priority of the recognition process according to the vehicle behavior and state. An example of changing the weighting according to the vehicle behavior and state will be described using Figures 9A to 9E.

9A to 9E, it is assumed that five modes are used as a method for weighting the priority of the recognition process, and the angle of view of the area for which the priority is increased for each mode is shown.

Mode DFF shown in Figure 9A is a high-speed cruising mode, and is used mainly when cruising at high speeds on expressways and other roads. When cruising at high speeds, the curves are gentle, so the area where priority is increased can be narrowed to the front. However, since distant objects are likely to approach in a short period of time, recognition processing is required for high-resolution images in order to recognize distant objects, and the frequency of recognition processing should also be increased. Therefore, the area where priority is increased for recognition processing is limited to the camera angle of view F410.

Mode DF shown in Fig. 9B is a forward mode, and is mainly used when cruising on public roads. When driving on public roads, it is not necessary to recognize objects farther away than when cruising at high speeds. However, it is necessary to deal with sudden sudden changes in the direction of the vehicle from the left and right and sharp curves. Therefore, in addition to the camera angle of view F410, the recognition priority for the camera angle of view FL420 and the camera angle of view FR430 is also increased.

Mode DR shown in Fig. 9C is a mode for reverse driving, and is the mode when the reverse gear is engaged. When reversing, the driving speed is slow, but the steering angle is often large when parking in a garage, etc. Therefore, in addition to camera view angle R460, which is the view angle for rearward monitoring, the recognition priority is also increased for camera view angles RL440 and RR450, which are mainly used for diagonal rearward monitoring.

Mode DFL shown in Fig. 9D is a mode used when the steering angle is turned left by a certain amount or more, other than when reversing, and when the left turn signal is on. When the steering angle is turned left by a certain amount or more, or when the left turn signal is on, it is expected that the vehicle will turn left or change lanes to the left lane. Therefore, in addition to camera view angle F410, which covers the front, camera view angle FL420, which covers the left front that is expected to be the destination, and camera view angle RL440, which is used to detect an object approaching from the rear left or an object that may be hit by the vehicle when turning left, are also given higher recognition priority.

Mode DFR shown in FIG. 9E is a mode when the steering angle is turned to the right by a certain amount or more in a state other than reverse, and when the right turn signal is on, and is the left-right opposite mode of Mode DFL.

When increasing the priority of the recognition process, if the angle of view for which the priority is to be increased matches the angle of view of each camera, as shown in Figures 9A to 9E, the priority of the recognition process can be increased on an image data basis acquired by each camera.

However, if there is a margin in the processing performance of the ECU, for example, in Mode DF shown in Fig. 9B, it is also possible to increase the priority of the recognition process for the area of the images acquired by camera RL 340 and camera RR 350 that overlaps with the camera view angles FL 420 and FR 430. Even if an object is in the same direction as seen from vehicle 10, by performing recognition processing on multiple image data acquired by multiple cameras, it is possible to recognize an object that is in a blind spot for one of the cameras and to improve recognition accuracy by comparing the recognition results for each image.

Strictly speaking, the recognition process for image data acquired by the cameras is a recognition process for image data acquired by each camera that has been subjected to image correction processing. The same applies below.

When weighting the priority of the recognition process determined according to the behavior and state of the vehicle is taken into account in the priority of the recognition process determined from the collision grace time of each point cloud cluster based on the sensing results of the LiDAR F210 and the LiDAR R260, it is possible to treat the weighted priority result as the priority itself, compare it with the priority determined from the collision grace time, and treat the higher priority as the priority for the direction. In this case, the weighting of the priority of the recognition process determined according to the behavior and state of the vehicle is also divided into three levels: high, medium, and low, and the recognition priority is set to high, medium, or low corresponding to the weighting.

When weighting the priority of the recognition process is set to three levels, for example, in Mode DF, in the medium speed range, the camera view angle F410 is weighted high, and the camera view angle FL420 and the camera view angle FR430 are weighted medium, and in the low speed range, the camera view angle F410, the camera view angle FL420, and the camera view angle FR430 are all weighted high. In other words, in the medium speed range, since the vehicle is traveling on a main road where travel is possible at a certain speed, more emphasis is placed on forward monitoring, and in the low speed range, taking into account travel on residential roads and the like, recognition process is performed by uniformly increasing the priority for a wide view angle.

The priority of the recognition process required according to the vehicle's behavior and condition is weighted at three levels: high, medium, and low, and the priority of the recognition process using the collision grace period of the point cloud cluster is set to emergency, highest, high, medium, and low. If the higher priority is set as the priority for the direction, the high to low priority is selected based on the vehicle's behavior and condition as well as the collision grace period of the detected point cloud cluster, and the emergency and highest priorities are determined only by the collision grace period of the detected point cloud cluster.

That is, in the configuration of Example 2, prioritization of recognition processing using the collision grace time of point cloud clusters is performed only in the forward/backward direction, which can be sensed by LiDAR F210 and LiDAR R260, and therefore, urgent and highest priority are also generated only in the forward/backward direction. Although the host vehicle 10 moves in the forward/backward direction, including at high speeds, in the lateral direction, movement is limited to steering in normal situations where there is no slippage or the like, and since the movement speed is limited, there is little need for remote monitoring.

Therefore, the configuration of the second embodiment may be considered, taking into consideration the balance between the cost of the sensor and the processing load of the ECU.

Furthermore, if it is desired to more precisely determine the priority in the horizontal direction, it is possible to perform stereoscopic vision of the area where the angles of view overlap by combining cameras FL320 and RL340 in the left direction and cameras FR330 and RR350 in the right direction, and calculate the spatial position of each pixel from the distance information of each pixel obtained by stereoscopic vision and the position information of the pixel on the image, thereby treating each pixel as a point having spatial position information, and treating the set of these points as a point cloud. Once a point cloud is obtained, point cloud clusters can be obtained in the horizontal direction as well, and the collision grace time for each point cloud cluster can also be obtained, and the priority of the recognition process can be determined based on this information.

Furthermore, it is also possible to omit only LiDAR R260, or both LiDAR F210 and LiDAR R260, and widen the angle of view of the necessary cameras so that the surroundings of the vehicle 10 that cannot be sensed by LiDAR can be photographed with two or more cameras, extract a point cloud cluster from the distance information of each pixel obtained by stereoscopic vision using images acquired by multiple cameras, and determine the priority of the recognition process using the point cloud cluster and the collision grace time of each point cloud cluster.

In the second embodiment, the in-vehicle information processing device includes an ECU Type B120 (controller).

In Example 2, in addition to being able to achieve the same effect as Example 1, the priority of recognition processing is determined for the monitoring directions in the forward and backward directions where the relative speed between the vehicle 10 and other vehicles is likely to be high, and weighting is determined for the left and right sides according to the behavior and state of the vehicle 10, thereby reducing the cost of sensors required for point cloud acquisition and ensuring the calculation performance of the ECU by limiting Lidar point cloud processing to the forward and backward directions which are important as sensing targets.

Example 3
Next, a third embodiment of the present invention will be described.

10 shows an example of the connection configuration of a system using multiple ECUs with signal processing functions including recognition processing of data acquired from external sensors, based on the configuration shown in Example 2. The configuration of the multiple cameras and multiple LiDARs that are external sensors is the same as that of Example 2.

In the configuration shown in FIG. 10, the ECUs having signal processing functions including recognition processing are made up of a front sensing ECU 160 (first control unit), a rear sensing ECU 170 (second control unit), and an integrated ECU 150 (integrated control unit).

The front sensing ECU 160 is directly connected to the camera F310, the camera FL320, the camera FR330, and the LiDAR F210, and mainly performs external environment sensing processing in front of the host vehicle 10.

The rear sensing ECU 170 is directly connected to the camera R360, the camera RL340, the camera RR350, and the LiDAR R260, and mainly performs external environment sensing processing behind the vehicle 10.

Taking into consideration the amount of sensing data output per unit time from each external sensor, each external sensor is directly connected to the front sensing ECU 160 or the rear sensing ECU 170 .

The integrated ECU 150 is connected to the Front sensing ECU 160, the Rear sensing ECU 170, and other ECUs mounted on the vehicle 10 via a CAN or an in-vehicle Ethernet. In addition to integrating the results of external recognition processed by the Front sensing ECU 160 and the Rear sensing ECU 170, the integrated ECU 150 determines the priority of recognition processing for each direction of the vehicle 10 based on information about the point cloud clusters extracted by the Front sensing ECU 160 and the Rear sensing ECU 170 and on the behavior and state of the vehicle from other ECUs mounted on the vehicle 10, and makes adjustments to balance the processing load of the recognition processing of the Front sensing ECU 160 and the Rear sensing ECU 170.

The front sensing ECU 160 and the rear sensing ECU 170 are directly connected via an inter-ECU image transfer communication path 165, which is a high-speed communication path compatible with the transfer of moving image data acquired by multiple cameras. The inter-ECU image transfer communication path 165 uses, for example, high-speed serial communication. In this case, communication technology used for digitally connecting cameras and ECUs may be used. Even if the inter-ECU image transfer communication path 165 requires bidirectional communication of moving image data, it may be divided into two paths for data transfer from the front sensing ECU 160 to the rear sensing ECU 170 and data transfer in the reverse direction, and a unidirectional communication path may be used for each.

An example of a processing flow in the configuration shown in Fig. 10 is shown in Fig. 11. Fig. 11 shows the processing separately for the Front Sensing ECU 160, the Rear Sensing ECU 170, and the Integrated ECU 150. Each processing represents processing for one frame, and is repeated for each frame.

In the Front Sensing ECU processing performed by the Front Sensing ECU 160, the processing from the LiDAR F point cloud data acquisition step S810 to the LiDAR F point cloud cluster collision grace period calculation step S850 is the same processing as the LiDAR point cloud data acquisition step S710 to the collision grace period calculation step S750 described using Figure 5 in Example 1, and is performed on the point cloud data acquired by the LiDAR F210.

In the LiDAR F point cloud cluster collision grace period notification step S860, the range of the direction in which each point cloud cluster exists is calculated for each point cloud cluster, and this is notified to the integrated ECU 150 together with the collision grace period for that point cloud cluster. The notified data is referenced in the image recognition priority setting step S960 performed by the integrated ECU 150. The notification is performed using a communication path such as a CAN that connects the Front sensing ECU 160 and the integrated ECU 150.
Regarding orientation, in this embodiment 3, since images are acquired by multiple cameras with the horizontal direction divided, when performing recognition processing on image data acquired by each camera, only horizontal orientation information is required. However, when a configuration using a separate camera in the vertical direction is included, vertical information is also required. Furthermore, when images captured by individual cameras are divided horizontally and recognition processing is performed separately for the top and bottom, vertical information is also required to determine the priority of recognition processing for each divided image area.

In the LiDAR F point cloud cluster recognition processing step S870, a recognition process is performed on each point cloud cluster by referencing the point cloud data that constitutes the point cloud cluster. This recognition process detects the type of object and removes point cloud clusters that appear to be noise based on the spatial positional relationship of each point data that constitutes the point cloud cluster and the distribution of signal intensity of the reflected laser light. The recognition results are sent to the integrated ECU 150 in the recognition result integration processing step S980. The transmission is performed using a communication path such as a CAN that connects the Front Sensing ECU 160 and the integrated ECU 150.

In step S880 for setting image data transfer to the Rear sensing ECU 170, based on the result of step S970 for specifying the image transfer path mode of the integrated ECU 150, settings are made to transmit image data required by the Rear sensing ECU 170 to the Rear sensing ECU 170 via the inter-ECU image transfer compatible communication path 165, regarding image data acquired by the camera F310, camera FL320, and camera FR330 directly connected to the Front sensing ECU 160 and data generated based on such image data.

In S890 of acquiring F directly connected camera image data, image data is acquired from the camera F310, the camera FL320, and the camera FR330 that are directly connected to the Front sensing ECU 160.

The image data acquired in the F directly connected camera image data acquisition step S890 is corrected in the F direct camera image correction processing step S892 for image distortion, image brightness correction, color reproduction and correction, image misalignment, etc. At this time, corrections due to individual differences between each camera and errors in mounting the camera on the vehicle are also performed. Regarding color, when using image data acquired by an image sensor that uses color filters with different transmission wavelength characteristics such as red, green, and blue that vary depending on the pixel, color restoration is performed using multiple surrounding pixels.

In the direct camera image correction processing step S892, an image acquired from a certain camera may be divided into multiple image regions and processed as necessary. Also, different image region divisions may be performed for each camera. For example, when an image is acquired using a fisheye lens with a wide angle of view on a certain camera, distortion differs greatly between the center and periphery of the image. Therefore, it is conceivable to divide the image into a center region and multiple peripheries for that camera, cut out each region as a separate image, and perform appropriate distortion correction for each region.

Of the image data corrected in the F direct camera image correction processing step S892, the image data required by the Rear sensing ECU 170 is sent to the Rear sensing ECU 170 based on the settings made in the image data transfer setting step S880 to the Rear sensing ECU 170.

In the F directly connected camera image recognition processing step S900, recognition processing is performed on image data acquired by the camera F310, the camera FL320, and the camera FR330 that are directly connected to the Front sensing ECU 160, for image data that should be recognized and processed by the Front sensing ECU 160. The method, content, and processing order of each image data item depend on the result of the image recognition calculation resource amount determination step S965 by the integrated ECU 150.

The recognition processing results of the F directly connected camera image recognition processing step S900 are transmitted to the recognition result integration processing step S980 of the integrated ECU 150. The transmission is performed using a communication path such as a CAN that connects the Front sensing ECU 160 and the integrated ECU 150. Even for images acquired by a camera directly connected to the Front sensing ECU 160, recognition processing is not performed on image data for which recognition processing is left to the Rear sensing ECU 170.

In the F indirectly connected camera image data acquisition step S910, the image data required by the Front sensing ECU 160 among the image data acquired from the camera R360, the camera RL340, and the camera RR350 is received via the inter-ECU image transfer communication path 165.

In the F indirectly connected camera image recognition processing step S920, recognition processing is performed on the image data to be recognized from the image data received in the F indirectly connected camera image data acquisition step S910. The method and content of the recognition processing for each image data and the processing order for each image data are determined according to the result of the image recognition calculation resource amount determination step S965 by the integrated ECU 150.

The recognition processing result of the F indirectly connected camera image recognition processing step S920 is transmitted to the recognition result integration processing step S980 of the integrated ECU 150. The transmission is performed using a communication path such as a CAN that connects the front sensing ECU 160 and the integrated ECU 150.

The Rear sensing ECU processing performed by the Rear sensing ECU 170 is equivalent to the Front sensing ECU processing, except that the directly connected cameras are camera R360, camera RL340, and camera RR350, and the indirectly connected cameras (cameras directly connected to the Front sensing ECU 160) are camera F310, camera FL320, and camera FR330.

In the integrated ECU process performed by the integrated ECU 150, first, in a vehicle state acquisition step S950, information on the behavior and state of the vehicle 10 required for prioritizing and weighting the recognition process is acquired. The acquired behavior and state information includes the gear state (forward or reverse), the steering state, the turn signal illumination setting, etc.

In the image recognition priority setting step S960, the priority of the recognition processing for each image to be recognized is calculated using the methods shown in Examples 1 and 2, etc., based on the direction of the point cloud cluster and the collision grace period detected based on the sensing results from LiDAR F210 and LiDAR R260 obtained in the LiDAR F point cloud cluster collision grace period notification step S860 and the LiDAR R point cloud cluster collision grace period notification step S865, and the information obtained in the vehicle state acquisition step S950.

Possible images to be subjected to the recognition process include images acquired by each camera, images divided in association with image correction in the F directly connected camera image correction process step S892 or the R directly connected camera image correction process step S897, and images divided and handled when performing image recognition process. The image data unit for calculating the priority is determined depending on the unit of image data handled in the recognition process.

In the image recognition computation resource amount determination step S965, the computation resource amount for the recognition process and the order of the recognition process are determined for each image data of the recognition process based on the result of the image recognition priority setting step S960, in the same manner as in the recognition computation resource amount determination step S745 described in the first embodiment. However, consideration is also given to the division of the recognition process between the front sensing ECU 160 and the rear sensing ECU 170, and the amount and combination of image data that can be transferred via the inter-ECU image transfer communication path 165. The order of the recognition process is also determined for each of the front sensing ECU 160 and the rear sensing ECU 170.

The contents determined in the image recognition calculation resource amount determination step S965 are transmitted via a path such as CAN to the F directly connected camera image recognition processing step S900 and F indirectly connected camera image recognition processing step S920 performed by the Front sensing ECU 160, the R directly connected camera image recognition processing step S905 performed by the Rear sensing ECU 170, and the R indirectly connected camera image recognition processing step S925, and are reflected in each image recognition processing.

In the image transfer path mode designation step S970, based on the decision made in the image recognition calculation resource amount determination step S965, transfer setting information is transmitted to the image data transfer setting step S880 to the rear sensing ECU 170 and the image data transfer setting step S885 to the front sensing ECU 170 so that each ECU can refer to the image data required for the recognition processing.

As a result, based on the contents determined in the image transfer path mode designation step S970, the Front sensing ECU 160 and the Rear sensing ECU 170 transmit the image data acquired by the camera directly connected to each ECU that is required by the other ECU via the ECU-to-ECU image transfer compatible communication path 165.

In the recognition result integration processing step S980, the recognition results of the LiDAR F point cloud cluster recognition processing step S870, the F directly connected camera image recognition processing step S900, the F indirectly connected camera image recognition processing step S920, the LiDAR R point cloud cluster recognition processing step S875, the R directly connected camera image recognition processing step S905, and the R indirectly connected camera image recognition processing step S925 are integrated to obtain the final external environment detection result, which is provided for the driving plan and decision-making processing required for autonomous driving and driving assistance.

When integrating recognition results, recognition results from various directions relative to the vehicle 10 are synthesized, the consistency of recognition results from multiple external sensors is confirmed, and inconsistent recognition results are selected or excluded.

12 is a diagram showing an example of image data transfer settings made in image transfer path mode designation step S970 via the inter-ECU image transfer compatible communication path 165. Fig. 12 shows an example of transfer settings made in units of image data acquired by each camera.

In FIG. 12, only when a situation arises in which recognition processing has a high priority in one of the eight directions centered on the vehicle 10, image data is transferred via the inter-ECU image transfer compatible communication path 165, and recognition processing for images acquired by a camera that is not directly connected is performed by one or both of the front sensing ECU 160 and the rear sensing ECU 170.

The operation of each row shown in Fig. 12 will be described below. When the recognition processing priority for the front direction of the vehicle 10 is high, the Front sensing ECU 160 performs recognition processing only on image data acquired by the camera F310, and recognition processing for image data acquired by cameras other than the camera F310 is performed by the Rear sensing ECU 170. Therefore, the image data acquired by the camera FL320 and the camera FR330 (strictly speaking, data obtained by performing image correction processing on image data acquired by the cameras; the same applies hereinafter) is transmitted to the Rear sensing ECU 170 via the inter-ECU image transfer communication path 165.

When the recognition processing priority for the right front direction of the vehicle 10 is high, the Front sensing ECU 160 performs recognition processing only on image data acquired by the camera F310 and the camera FR330, and recognition processing for image data acquired by cameras other than the camera F310 and the camera FR330 is performed by the Rear sensing ECU 170. Therefore, the image data acquired by the camera FL320 is transmitted to the Rear sensing ECU 170 via the inter-ECU image transfer compatible communication path 165.

When the recognition processing priority for the right direction of the host vehicle 10 is high, the Front sensing ECU 160 performs recognition processing only on image data acquired by the cameras FR 330 and RR 350, and recognition processing for image data acquired by cameras other than FR 330 and RR 350 is performed by the Rear sensing ECU 170. Therefore, the image data acquired by the cameras F 310 and FL 320 is transmitted to the Rear sensing ECU 170 via the inter-ECU image transfer communication path 165. In addition, the image data acquired by the camera RR 350 is transmitted to the Front sensing ECU 160 via the inter-ECU image transfer communication path 165.

By collecting and processing image data acquired by camera FR330 and camera RR350, which have a high recognition priority, in Front sensing ECU 160, the entire right direction of vehicle 10 can be processed comprehensively by a single ECU, and there is an advantage that the consistency of the directions captured by both cameras can be confirmed by Front sensing ECU 160. Furthermore, in order to accommodate high-speed driving, it is conceivable that an ECU with higher processing performance than Rear sensing ECU 170 will be used as Front sensing ECU 160, and such a configuration has the advantage that recognition processing for directions with a high recognition priority can be performed by Front sensing ECU 160.

When the recognition processing priority for the right rear direction of the vehicle 10 is high, the Rear sensing ECU 160 performs recognition processing only on image data acquired by the camera R360 and the camera RR350, and recognition processing of image data acquired by cameras other than the camera R360 and the camera RR350 is performed by the Front sensing ECU 160. Therefore, the image data acquired by the camera RL340 is transmitted to the Front sensing ECU 160 via the inter-ECU image transfer compatible communication path 165.

When the recognition processing priority for the area behind the vehicle 10 is high, the Rear sensing ECU 170 performs recognition processing only on image data acquired by the camera R 360, and recognition processing for image data acquired by cameras other than the camera R 360 is performed by the Front sensing ECU 160. Therefore, the image data acquired by the camera RL 340 and the camera RR 350 is transmitted to the Front sensing ECU 160 via the inter-ECU image transfer compatible communication path 165.

If the recognition processing priority is high for the left front, left, and left rear directions of the vehicle 10, the directions are the same except for the difference between the left and right directions, i.e., the right front, right, and right rear directions, respectively.

In the third embodiment, as in the second embodiment, if it is desired to more strictly determine the priority in the horizontal direction, it is conceivable to combine cameras FL320 and RL340 in the left direction and cameras FR330 and RR350 in the right direction, and use stereo vision for the areas where the angles of view overlap.

Since stereo vision requires reference to both sets of image data acquired by the associated cameras, it is better to collect and process the image data from the associated cameras in one ECU. Therefore, when performing processing related to left-right stereo vision, an ECU that will perform processing to obtain a point cloud cluster from the disparity information of each pixel of the image data is determined in advance, and the usage settings for the inter-ECU image transfer compatible communication path 165 are made so that the ECU can reference the image data required for processing related to the point cloud cluster using stereo vision.

FIG. 13 shows the usage settings of the inter-ECU image transfer communication path 165 required for performing stereoscopic viewing in the left and right directions. In FIG. 13 , when the Front sensing ECU 160 performs stereoscopic viewing and converts the results into point cloud cluster information for the right direction, image data acquired by camera RR 350 is always transmitted from the Rear sensing ECU 170 to the Front sensing ECU 160 via the inter-ECU image transfer communication path 165. When the Rear sensing ECU 170 processes stereoscopic viewing for the right direction and obtains point cloud cluster information from the results, image data acquired by camera FR 330 is always transmitted from the Front sensing ECU 160 to the Rear sensing ECU 170 via the inter-ECU image transfer communication path 165. The same applies to image data acquired by cameras FL 320 and RL 340 for the left direction.

The ECU that obtains point cloud cluster information from the stereo vision and the results thereof may be the Front sensing ECU 160 or the Rear sensing ECU 170 for both the left and right directions, depending on the configuration of the Front sensing ECU 160 and the Rear sensing ECU 170. Also, different ECUs may process the left direction by the Front sensing ECU 160 and the right direction by the Rear sensing ECU 170. It is also possible to dynamically allocate processing taking into account the priority of image recognition processing.

Because the acquisition of point cloud cluster information is required in image recognition priority calculation step S960, the stereoscopic vision and the process of obtaining point cloud cluster information from the results must be performed before image recognition priority calculation step S960. Therefore, the creation of point cloud cluster information using stereoscopic vision is performed using image data from the previous frame. Alternatively, the ECU that performs the stereoscopic vision and the process of obtaining point cloud cluster information from the results may be fixed, and the camera image data required for the stereoscopic vision processing may be constantly sent to the ECU, and the processes up to the stereoscopic vision processing may be performed at an early stage of processing, separate from the image recognition processing. Stereoscopic vision itself is basically performed by local matching processing of paired images after image correction processing, so it can be performed at a stage prior to the recognition processing.

In the third embodiment, the in-vehicle information processing device includes a front sensing ECU 160 (first control unit) and a rear sensing ECU 170 (second control unit).

In Example 3, it is also possible to omit only LiDAR R260, or both LiDAR F210 and LiDAR R260, and use two or more cameras to capture the area around the vehicle 10 that cannot be sensed by LiDAR, for example by widening the angle of view of the necessary cameras, obtain a point cloud using distance information calculated from the parallax information of each pixel in stereoscopic vision of the image data obtained by the cameras, extract point cloud clusters from the point cloud, calculate the collision grace time for each point cloud cluster, and determine the priority of the recognition process.

Except for the left and right directions, the cameras that acquire the image data required for stereo vision are directly connected to the same ECU. Therefore, if the ECU performs the stereo vision and the processing that determines point cloud cluster information from the results, there is no need to transfer image data via the inter-ECU image transfer compatible communication path 165 associated with the stereo vision processing.

According to the third embodiment, the vehicle is equipped with a Front sensing ECU 160 and a Rear sensing ECU 170, and is configured so that recognition processing is performed by one ECU that is suitable for processing output from a camera relating to a high-priority recognition direction, and recognition processing is performed by another ECU for output from a camera relating to directions other than the high-priority recognition direction. This makes it possible to realize an in-vehicle information processing device, an in-vehicle information processing system, and an in-vehicle information processing method that can further reduce recognition delays for objects with a high level of urgency.

Example 4
Next, a fourth embodiment of the present invention will be described.

14 shows a configuration used in a relatively simple driving assistance system or the like as Example 4. Example 4 has only a camera F310 and a LiDAR F210 as sensors for sensing the external world, and includes an ECU Type C130 (controller) having a function of recognizing the external world based on information obtained from these sensors.

The ECU Type C130 communicates with other ECUs installed in the vehicle 10 via an interface such as a CAN. By connecting with other ECUs, it is possible to transmit the results of recognizing the outside world to ECUs that make driving decisions and control driving, and to obtain the driving status of the vehicle.

In the fourth embodiment, image data acquired by the camera F310 is subjected to image correction processing such as image distortion correction, and then the image data is divided into three areas, camera angle of view FA411, camera angle of view FB412, and camera angle of view FC413, as shown in Fig. 15. These areas correspond to areas A511, B512, and C513 of the image of the camera F shown in Fig. 6.

Using point cloud data acquired by the LiDAR F210, point cloud clusters are extracted in the same manner as in Examples 1 to 3 described above, the collision grace time for each point cloud cluster is obtained, and the priority of recognition processing for each direction of the camera view angle FA411, the camera view angle FB412, and the camera view angle FC413 is calculated based on the direction in which each point cloud cluster is located. In addition, recognition processing is performed for each point cloud cluster.

The point cloud data used to create the point cloud cluster may be data obtained by sensing with millimeter wave radar or stereo vision with a camera, as in the above-described Examples 1 to 3. In particular, when stereo vision with a camera is used, the configuration shown in Fig. 14 can be modified to a configuration in which only one stereo camera is connected as an external sensor.

The weighting of the recognition process priority may also be performed based on the behavior or state of the vehicle, as in the first to third embodiments described above.

As shown in FIG. 6, the recognition process is performed on the image data of each of the areas A511, B512, and C513 in accordance with the method according to the final recognition process priority that reflects the weighting of the recognition process priority.

The recognition results based on the image data of each area and the point cloud data acquired by LiDAR undergo a recognition result integration process to become the final external recognition results, which are used to process driving plans and decisions required for driving assistance.

Even with a relatively simple external sensor configuration, as in Example 4 shown in Figure 14, by dividing the image acquired by the camera into multiple areas, recognition processing with adjusted priority according to urgency is possible.

By using the relative speed with respect to surrounding objects, even if an object is in the expected travel area, an object moving away from the vehicle 10, i.e., an object with little need to be recognized, can be excluded from the priority objects, thereby reducing the amount of calculation required for the recognition process. Furthermore, by determining the priority of the recognition process using the time until collision, it is possible to increase the recognition priority of an object with little time until collision even in cases where the object is approaching from a surrounding object, and to reduce delays in recognition of objects with high urgency.

In other words, even when there are many objects around the vehicle 10 or when there are fast-moving objects such as bicycles or automobiles, the priority for recognition processing can be set to appropriately reflect the necessity and urgency, making it possible to perform recognition processing efficiently.

In the fourth embodiment, the in-vehicle information processing device includes an ECU Type C (controller).

In the fourth embodiment, the same effects as those in the first embodiment can be obtained.

In this specification, LiDAR has been mainly used as a means for measuring the distance from the vehicle 10 to an object and detecting the relative speed between the vehicle 10 and the object, but in the present invention, millimeter-wave radar or stereo vision using multiple cameras may also be used.

Furthermore, even if the external world sensing is done by means other than LiDAR, millimeter wave radar, or stereo vision using multiple cameras, as long as the means can acquire at least the presence, direction, and distance of an object before the recognition process, the priority of the recognition process can be determined based on the content of these information and the direction of external world sensing, so the sensing means can be any.

Furthermore, the ECU Type A110 shown in FIG. 4 includes an image correction processing unit 116, an image recognition processing unit 117, and a point cloud cluster recognition processing unit 118. However, the present invention can be implemented even if the image correction processing unit 116 is omitted, or even if either the image recognition processing unit 117 or the point cloud cluster recognition processing unit 118 is omitted.

Furthermore, it is also conceivable to perform processing similar to recognition processing as a means for measuring the distance from the vehicle 10 to an object or detecting the relative speed and distance between the vehicle 10 and the object. If the processing similar to recognition processing as a means for detecting the relative speed and distance is a simple recognition processing that is performed with a lighter load than the recognition processing on sensing data that is performed after the processing, it is possible to perform the final recognition processing efficiently. Here, the sensing data that is the target of the recognition processing does not need to be limited to camera images, and processing on sensing data obtained by other external sensors may also be performed.

The external sensor configurations and ECU configurations shown in this specification are examples, and other configurations may be used as long as they can adjust the priority of recognition processing based on the techniques shown in this specification. The processing in the ECU may be implemented using any method, such as software, electronic circuits such as logic circuits, or a combination of software and electronic circuits. Furthermore, the ECU may not only be packaged in a single box, but may also be in any form that can realize the function, such as a SoC (System on Chip) or a board with electronic circuits mounted on it.

DESCRIPTION OF SYMBOLS 10...Own vehicle, 20...Vehicle A, 30...Vehicle B, 40...Vehicle C, 110...ECU Type A, 111...External data acquisition unit, 112...Point cloud cluster generation unit, 113...Priority setting unit, 114...Computational resource amount determination unit, 115...Recognition result integration processing unit, 116...Image correction processing unit, 117...Image recognition processing unit, 118...Point cloud cluster recognition processing unit, 120...ECU Type B, 130...ECU Type C, 150...Integrated ECU, 160...Front sensing ECU, 165...Inter-ECU image transfer compatible communication path, 170...Rear sensing ECU, 210...LiDAR F, 260...LiDAR R, 290...LiDAR A, 310...Camera F, 320...Camera FL, 330...Camera FR, 340...Camera RL, 350...Camera RR, 360...Camera R, 410...Camera angle of view F, 411...Camera angle of view FA, 412...Camera angle of view FB, 413...Camera angle of view FC, 420...Camera angle of view FL, 430...Camera angle of view FR, 440...Camera angle of view RL, 450...Camera angle of view RR, 460...Camera angle of view R, 511...area A, 512...area B, 513...area C, 514...area D, 515...area E, 516...area F, 517...area G, 611...[A] (processing of area A), 612...[B, A, C] (processing of areas B, A and C), 616...[D] (processing of area D), 617...[E] (processing of area E), 618...[D, F] (processing of areas D and F), 619...[E, G] (processing of areas E and G)

Claims (12)

an external data acquisition unit that acquires external data from an external sensor mounted on the vehicle;
a point cloud cluster generation unit that performs clustering on a plurality of points constituting the external world data acquired from all or a part of the external world sensors to generate a point cloud cluster;
a priority setting unit that sets a priority to the point cloud cluster or an external environment area in which the point cloud cluster exists, based on a relative speed of the point cloud cluster with respect to the vehicle calculated from the external environment data related to the point cloud cluster;
a computational resource amount determination unit that determines the amount of computational resources for the recognition process of the external world data in accordance with the priority;
an image correction processing unit that corrects an image of the outside world data acquired by the outside world data acquisition unit;
an image recognition processing unit that recognizes the image corrected by the image correction processing unit based on the amount of calculation resources determined by the calculation resource amount determination unit;
a point cloud cluster recognition processing unit that performs object type detection and noise removal on the point cloud cluster generated by the point cloud cluster generation unit, and performs a recognition process on the point cloud cluster;
a recognition result integration processing unit that checks the consistency of the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit, selects or excludes inconsistent recognition results, integrates the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit, and outputs the integrated result as an external world recognition result;
1. An in-vehicle information processing device comprising: a control unit having: 2. The information processing device according to claim 1,
The priority setting unit uses the relative speed to calculate a collision grace period that represents the time required for the point cloud cluster to come into contact with the vehicle when it is assumed that the point cloud cluster will come into contact with the vehicle, and sets the priority to the point cloud cluster or an external area in which the point cloud cluster is located based on the collision grace period. 3. The in-vehicle information processing device according to claim 1,
10. An in-vehicle information processing device, wherein all or some of the external sensors are cameras. 4. The in-vehicle information processing device according to claim 3,
The in-vehicle information processing device is characterized in that the computational resource amount determination unit determines the computational resource amount for the recognition process for each image area in the image captured by the camera according to the priority. 4. The in-vehicle information processing device according to claim 3,
The in-vehicle information processing device is characterized in that the computational resource amount determination unit changes at least one of the resolution of the image captured by the camera, the cycle of the recognition processing, and the logic of the recognition processing according to the priority. 4. The in-vehicle information processing device according to claim 3,
The external environment data acquisition unit acquires a plurality of images captured by a plurality of the cameras, and the computational resource amount determination unit determines the amount of computational resources for recognition processing for each of the plurality of images according to the priority. 4. The in-vehicle information processing device according to claim 3,
The external environment data acquisition unit acquires a plurality of images captured by a plurality of the cameras, and the computational resource amount determination unit determines the amount of computational resources for recognition processing for each image area in the plurality of images according to the priority. 3. The in-vehicle information processing device according to claim 1,
The in-vehicle information processing device, characterized in that the external sensor has a camera and a LiDAR. 3. The in-vehicle information processing device according to claim 1,
The in-vehicle information processing device, characterized in that the external sensor includes a camera, a LiDAR for monitoring the front of the vehicle, and a LiDAR for monitoring the rear of the vehicle. 3. The in-vehicle information processing device according to claim 1,
The control unit
a first control unit that senses an external environment in front of the vehicle;
a second control unit that senses the external environment behind the vehicle;
and
The in-vehicle information processing device, characterized in that the external sensor includes a camera, a LiDAR for monitoring the front of the vehicle, and a LiDAR for monitoring the rear of the vehicle. a plurality of external sensors that detect objects around the vehicle;
A control unit;
Equipped with
The control unit
an external data acquisition unit that acquires external data from the plurality of external sensors;
a point cloud cluster generation unit that performs clustering on a plurality of points constituting the external world data acquired from all or a part of the external world sensors to generate a point cloud cluster;
a priority setting unit that sets a priority to the point cloud cluster or an external environment area in which the point cloud cluster exists, based on a relative speed of the point cloud cluster with respect to the vehicle calculated from the external environment data related to the point cloud cluster;
a computational resource amount determination unit that determines the amount of computational resources for the recognition process of the external world data in accordance with the priority;
an image correction processing unit that corrects an image of the outside world data acquired by the outside world data acquisition unit;
an image recognition processing unit that recognizes the image corrected by the image correction processing unit based on the amount of calculation resources determined by the calculation resource amount determination unit;
a point cloud cluster recognition processing unit that performs object type detection and noise removal on the point cloud cluster generated by the point cloud cluster generation unit, and performs a recognition process on the point cloud cluster;
a recognition result integration processing unit that checks the consistency of the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit, selects or excludes inconsistent recognition results, integrates the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit, and outputs the integrated result as an external world recognition result;
An in-vehicle information processing system comprising: Detecting objects around the vehicle using multiple external sensors;
Acquire external world data from the plurality of external world sensors;
The control unit of the in-vehicle information processing device
clustering a plurality of points constituting the external world data acquired from all or a part of the external world sensors to generate a point cloud cluster;
setting a priority for the point cloud cluster or an external environment area in which the point cloud cluster exists based on a relative speed of the point cloud cluster with respect to the vehicle calculated from the external environment data related to the point cloud cluster;
determining an amount of computational resources for the recognition processing of the external world data according to the priority ;
Correcting the acquired image of the external world data;
Recognizing the corrected image based on the amount of computational resources;
performing object type detection and noise elimination on the point cloud cluster, and performing recognition processing on the point cloud cluster;
The recognition processing result of the point cloud cluster and the recognized corrected image are checked for consistency with each other, and inconsistent recognition results are selected and excluded, and the recognition processing result of the point cloud cluster and the recognized corrected image are integrated and output as an external world recognition result.
10. An in-vehicle information processing method comprising: