JP7380904B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

Patent Document 1 describes providing a surrounding environment recognition device capable of early detection of a moving three-dimensional object even in a situation where a moving three-dimensional object is apparently overlapping with another three-dimensional object. When the surrounding environment recognition device captures multiple images in chronological order, it detects three-dimensional objects based on the distance from the aircraft, and uses the multiple images to track feature points inside a predetermined area containing the three-dimensional object. By doing so, the motion vector of the feature point is detected, and based on the detection result, a moving three-dimensional object that exists inside the area is detected.

Patent Document 2 relates to an information processing device, an information processing method, a program, a moving object control device, and a moving object that improve recognition accuracy of a target object. The information processing device converts at least one of a photographed image obtained by the image sensor and a sensor image representing a sensing result of a sensor whose sensing range overlaps at least a portion of the image sensor, and converts the coordinates of the photographed image and the sensor image. The systems are matched, and object recognition processing is performed based on the photographed image and sensor image with the matched coordinate systems.

International Publication No. 2020/138286 International Publication No. 2020/116195

It tends to be difficult to detect objects that are far away or in situations where visibility is poor. On the other hand, depending on usage conditions, it may be expected to reliably detect objects that are far away from the device itself or in situations where visibility is poor. For example, when driving on a highway or under bad weather such as dense fog or rain, it is important to detect objects in the direction of travel of the vehicle as early as possible in order to ensure safety while driving.

One object of the present invention is to provide an information processing method, an information processing method, and a program that solve the above problems.

According to a first aspect of the present invention, an information processing device includes an object detection unit that detects objects around the device, and an object determination unit that determines the type of the object, and the object detection unit , an object recognition unit, an area recognition unit, and a depth information processing unit, and the object recognition unit determines whether an object is represented in the image from image data representing the image using signal values of each of a plurality of pixels. the depth information processing unit detects a first region that is a region that corresponds to the pixel, and determines a first reliability that is the reliability of a first type that is the type of object represented in the first region; The area recognition unit divides the image into second areas representing different types of objects based on the image data, and divides the image into second areas where different types of objects are represented. A second type, which is the type of object represented in the second area, is determined for each area, and the object determining unit is configured to maintain the depth of the object represented in the second area within a predetermined range. adjusting the second area, determining the type of object in a non-overlapping area that does not overlap with the first area in the second area as a second type, and determining that the type of object in the second area overlaps with the first area If the first reliability is less than a predetermined reliability threshold in the overlapping area, the type of object in the overlapping area is determined to be a second type, and in the overlapping area, the first reliability is less than the predetermined reliability threshold. If the degree is equal to or greater than the threshold, the type of object in the overlapping area is determined to be the first type.

According to a second aspect of the present invention, there is provided an information processing method in which an information processing apparatus calculates a region representing an object in the image from image data representing an image using signal values of each of a plurality of pixels. a first step of detecting a first region and determining a first reliability that is a first type of reliability that is the type of object represented in the first region; and detecting the depth of the object for each sample point corresponding to the pixel. a second step of obtaining depth information indicating the image data; dividing the image into second regions in which different types of objects are represented; a third step of determining a second type of object represented in the second area, and a fourth step of adjusting the second area so that the depth of the object represented in the second area is maintained within a predetermined range; The type of object in a non-overlapping area that does not overlap with the first area in the second area is determined as a second type, and the first reliability is determined in an overlapping area that overlaps with the first area in the second area. If the degree is less than a predetermined reliability threshold, determine the type of object in the overlapping region as a second type, and in the overlapping region, if the first reliability is equal to or higher than the predetermined reliability threshold, A fifth step of determining the type of object in the overlapping area as the first type is executed.

In a third aspect of the present invention, the program includes, in a computer, an object detection unit that detects objects around the device, and an object determination unit that determines the type of the object, and the object detection unit includes: The object recognition unit includes an object recognition unit, a region recognition unit, and a depth information processing unit, and the object recognition unit is configured to represent an object in the image from image data representing the image using signal values of each of a plurality of pixels. detecting a first region that is a region, determining a first reliability that is a reliability of a first type that is a type of object represented in the first region, and determining a first reliability that is a reliability of a first type that is a type of object represented in the first region; Depth information indicating the depth of the object is acquired for each sample point, and the area recognition unit divides the image into second areas representing different types of objects based on the image data, and divides the image into second areas representing different types of objects. The object determination unit determines a second type of the object represented in the second area for each case, and the object determination unit determines the second type of the object represented in the second area so that the depth of the object represented in the second area is maintained within a predetermined range. adjusting a second area , determining the type of object in a non-overlapping area that does not overlap with the first area in the second area as a second type; If the first reliability in the overlapping area is less than a predetermined reliability threshold, the type of object in the overlapping area is determined to be a second type, and in the overlapping area, the first reliability is lower than the predetermined reliability. , the information processing apparatus operates as an information processing apparatus that determines the type of object in the overlapping area as the first type.

According to the present invention, objects around the device can be detected more reliably.

1 is a schematic block diagram showing a configuration example of an information processing device according to an embodiment. FIG. FIG. 2 is a schematic block diagram showing an example of a functional configuration of an information detection section according to the present embodiment. It is a figure showing an example of image data. FIG. 3 is a diagram showing regions detected by the object recognition unit according to the present embodiment. It is a figure showing the area divided by the area recognition part concerning this embodiment, and the area detected. FIG. 3 is a diagram showing a depth image showing depth acquired by the depth information processing unit according to the present embodiment. 7 is a flowchart illustrating an example of object detection processing according to the present embodiment. FIG. 3 is a state transition diagram showing an example of object detection control according to the present embodiment. FIG. 2 is a schematic block diagram illustrating the minimum configuration of this embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a schematic block diagram showing a configuration example of an information processing device 10 according to an embodiment of the present invention. In the example shown in FIG. 1, the information processing device 10 is configured as a driving control device and forms part of the vehicle control system 1. The vehicle control system 1 is mainly installed in a vehicle and used for the operation of the vehicle, but when it is implemented, some or all of the system may not be installed in the vehicle. In this application, the vehicle in which the vehicle control system 1 and the information processing device 10 are installed may be referred to as the own vehicle, the information processing device 10 as the own device, and the component to be explained or processed as the own part.

The information processing device 10 according to the present embodiment uses image data representing an image using signal values of each of a plurality of pixels, and depth information representing the depth of an object for each sample point corresponding to a pixel to display the image in an image. Detect objects that are
The information processing device 10 detects a region where an object is represented from the image as a first region, determines the type of object represented in the first region, and determines the reliability of the determined object type as the first reliability. . On the other hand, the information processing device 10 divides the image into regions where different types of objects are represented as second regions, and determines the type of object represented in the second region for each second region. Based on the depth information indicating the depth of the object for each sample point corresponding to each pixel, the information processing device 10 controls the second area so that the depth of the object represented in the second area is maintained within a predetermined range. Two areas may be adjusted.
The information processing device 10 determines that the type of object in a non-overlapping area that does not overlap with the first area is the second type, out of the determined second area, and determines that the first reliability is a predetermined level in the overlapping area that overlaps with the first area. is less than the reliability threshold, the object type is determined to be the second type. The information processing device 10 determines that the type of object in the overlapping area is the first type when the first reliability is equal to or higher than a predetermined reliability threshold in the overlapping area.

The region of the object to be determined is information indicating the position and size of the object, and the type of the object to be determined constitutes part of the information regarding the situation around the own vehicle.
As described below, the vehicle control system 1 detects the situation of the own vehicle as well as the surrounding situation of the own vehicle, and controls the operation of the own vehicle using the detected information. In the example shown in FIG. 1, the information processing device 10 functions as a driving control unit that performs processing related to autonomous driving or driving support using information regarding the detected situation of the own vehicle and the surrounding situation. More specifically, the information processing device 10 implements functions such as collision avoidance of the own vehicle, shock mitigation, follow-up driving while maintaining the following distance, vehicle speed maintenance driving, collision warning of the own vehicle, lane departure warning, etc. Execute processing. Further, the information processing device 10 may execute processing for autonomous driving without relying on the driver's operation.

The vehicle control system 1 includes an operation section 101, an information collection section 102, a communication section 103, an in-vehicle device 104, an output control section 105, an output section 106, a drive system control section 107, a drive system 108, a vehicle system control section 109, and a vehicle system. 110, a storage section 111, and components such as the information processing device 10. Each component is wirelessly or wired connected to other components using the network NW so that various data can be mutually transmitted. The network NW includes, for example, a communication network such as a CAN (Controller Area Network) and a LAN (Local Area Network) that conforms to a predetermined standard, a data bus, and the like.

The operation unit 101 receives operations from a user such as a passenger on the vehicle, and inputs instructions and various data according to the received operations. The operation unit 101 generates an input signal based on input data, instructions, etc., and supplies the generated input signal to each component of the vehicle control system 1. The operation unit 101 includes an operation device that accepts manual operation, such as a button, switch, lever, or touch panel. The operation unit 101 may include an operation device that detects an operation by means other than manual operation, such as a voice command from an audio signal input from a microphone inside the car or a gesture from an image input from a camera inside the car. Further, the operation unit 101 does not necessarily need to include an operation device. The operation unit 101 may include an input interface that receives an input signal from an operation device wirelessly or by wire.

The information collection unit 102 collects various types of information used in various processes by the vehicle control system 1. The information collection unit 102 includes various sensors for collecting such information. The information collection unit 102 supplies the collected information to each component of the vehicle control system 1. The information collection unit 102 includes various sensors for detecting environmental information around the host vehicle. More specifically, the information collecting unit 102 includes, as an example of a sensor for detecting surrounding objects, a camera, a distance measuring sensor, etc., or a combination thereof. The information collecting unit 102 includes one or a combination of a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, etc., as an example of a sensor for detecting the surrounding weather. The information collecting unit 102 includes a GNSS (Global Navigation Satellite System) receiver as an example of a sensor for detecting the current position of the host vehicle. A GNSS receiver receives GNSS signals from each of four or more satellites orbiting around the earth, and estimates its own position based on the time difference between the received GNSS signals. The information collection unit 102 includes a sensor for detecting the driving state of the own vehicle. More specifically, the information collection unit 102 includes a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU), an operation amount of an accelerator pedal, an operation amount of a brake pedal, a steering angle of a steering wheel, and an engine rotation speed. , a combination of sensors for detecting motor rotation speed, wheel rotation speed, etc. Additionally, the information collection unit 102 includes various sensors for detecting information within the own vehicle. More specifically, the information collection unit 102 includes a camera that photographs passengers including the driver, a microphone that collects sounds inside the vehicle, and the like.

The communication unit 103 communicates with the in-vehicle equipment 104 and external equipment that is equipment other than the vehicle control system 1, and transmits and receives various data to and from other parts of the vehicle control system 1 wirelessly or by wire. The communication unit 103 may be able to communicate with devices connected to the Internet, a public network, or a private network via a base station or access point that constitutes an external network. This makes it possible to realize V2X communications such as vehicle-to-vehicle communications, vehicle-to-infrastructure communications, vehicle-to-pedestrian communications, and vehicle-to-home communications. The communication unit 103 may also include a beacon communication unit. The beacon communication unit receives radio waves or electromagnetic waves transmitted from a wireless station installed on the road, and receives information such as the location, traffic congestion, traffic regulations, required time, etc. information may be received.

The in-vehicle device 104 is a variety of information devices installed or carried within the vehicle. The in-vehicle device 104 corresponds to a navigation device that searches for and guides a route from a departure point or current position to a destination point, a mobile phone owned by a passenger, and the like.

The output control unit 105 controls the output of various types of information to the occupants of the own vehicle or to the outside of the own vehicle. The output control unit 105 controls the generation of at least one of visual information such as image data, auditory information such as audio data, and supply of the generated information to the output unit 106 . More specifically, when there is a possibility that an event such as a collision, a collision, or entry into an area where danger is expected to occur, the output control unit 105 generates a warning sound or a warning message guidance voice for the event. The generated audio data is supplied to the output unit 106 as an output signal.

The output unit 106 outputs various information regarding the occupant of the own vehicle or the outside of the own vehicle. The output unit 106 includes, for example, a liquid crystal display, an instrument panel, an audio speaker, a projector, a lamp, and the like.

The drive system control unit 107 controls the drive system 108 of the own vehicle. Drive system control section 107 generates various control signals and supplies the generated control signals to drive system 108 . Drive system control section 107 may output control signals to components other than drive system 108. As a result, the output destination component is notified of the control state of the drive system 108.

The drive system 108 is configured to include various devices related to driving the own vehicle. The drive system 108 includes, for example, an internal combustion engine, a driving force generating device such as a driving motor, a driving force transmission mechanism for transmitting the generated driving force to the wheels, and a steering wheel for adjusting the steering angle. It is equipped with a mechanism, a braking device to generate braking force, ABS (Antilock Brake System), ESC (Electric Stability Control), a power steering device, etc.

A vehicle system control unit 109 controls a vehicle system 110 of the own vehicle. Vehicle system control section 109 generates various control signals and supplies the generated control signals to vehicle system 110. Vehicle system control section 109 may output control signals to components other than vehicle system 110. As a result, the output destination component is notified of the control state of the vehicle system 110.

The vehicle system 110 is configured to include various devices that make up the vehicle system of the host vehicle. The vehicle system 110 includes, for example, a keyless entry system, a smart key system, a power window, a power seat, a steering wheel, an air conditioner, and various lamps. Examples of the various types of lamps include headlamps, back lamps, brake lamps, blinkers, fog lamps, and the like.

The storage unit 111 stores various programs used by each component of the vehicle control system 1, various data, or various acquired data. The storage unit 111 includes various storage devices such as ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive).

The information processing device 10 includes an information detection section 11 , a position estimation section 12 , a situation analysis section 13 , a motion planning section 14 , and a motion control section 15 . The information processing device 10 may include, for example, a control device such as an ECU (Electronic Control Unit). The information processing device 10 includes a processor such as a CPU (Central Processing Unit), and the processor executes processing instructed by various commands written in the program as processing using a predetermined program to detect information. The functions of the unit 11, the position estimation unit 12, the situation analysis unit 13, the motion planning unit 14, and the motion control unit 15 may be realized. In the following description, executing processes instructed by various commands written in a program may be referred to as "program execution", "executing a program", etc.

The information detection unit 11 detects various types of information required for vehicle driving control. The information detection section 11 detects, for example, information external to the own vehicle based on data from each section of the vehicle control system 1. The information detection unit 11 performs, for example, detection processing, recognition processing, tracking processing, and distance detection processing of objects around the host vehicle. Examples of objects to be detected include other vehicles, people, obstacles, structures, roads, traffic lights, traffic signs, and road markings. The information detection unit 11 may detect the environment around the own vehicle. The information detection unit 11 detects, for example, the environment around the host vehicle. The surrounding environment to be detected includes, for example, weather, temperature, humidity, brightness, and road surface conditions.

The information detection section 11 may detect information external to the own vehicle based on data from each section of the vehicle control system 1. The information detection unit 11 performs driver authentication processing and recognition processing, detection of the driver's condition, passenger detection, and detection of the environment inside the vehicle. The driver's condition to be detected may include, for example, concentration level, fatigue level, line of sight direction, and the like. The environment inside the vehicle to be detected may include, for example, temperature, humidity, brightness, and the like.

The information detection section 11 may detect the state of the own vehicle based on data from each section of the vehicle control system 1. The state of the own vehicle to be detected includes, for example, speed, acceleration, steering angle, presence or absence of an abnormality, if an abnormality is detected, its content, driving operation status, position and tilt of the power seat, door lock. and the status of other in-vehicle equipment.
The information detection section 11 outputs detection data indicating the detection result to the position estimation section 12, the situation analysis section 13, and the operation control section 15. A configuration example of the information detection unit 11 will be described later.

The position estimation unit 12 estimates the position and orientation of the own vehicle based on data from each part of the vehicle control system 1, such as the information collection unit 102, the information detection unit 11, and the situation analysis unit 13. The position estimating unit 12 also generates map information (hereinafter referred to as a self-position estimation map) used for estimating the self-position. The position estimation unit 12 applies, for example, SLAM (Simultaneous Localization and Mapping) when generating the self-position estimation map. The position estimating unit 12 outputs position data indicating the position and orientation of the own vehicle, which is the estimation result, to the situation analyzing unit 13. The position estimation unit 12 stores the generated self-position estimation map in the storage unit 111.
Note that the position estimation section 12 may output data indicating the recognition result to the information detection section 11. The information detection unit 11 may use data input from the position estimation unit 12 in order to detect information external to the own vehicle.

The situation analysis unit 13 analyzes the situation of the own vehicle and the surroundings of the own vehicle.
The situation analysis unit 13 analyzes various types of map information stored in the storage unit 111 based on data from each component of the vehicle control system 1 such as the position estimation unit 12 and the information detection unit 11, and performs driving control. Generate map information that includes information used for. The situation analysis unit 13 outputs the generated map information to the motion planning unit 14 and the like.
The situation analysis unit 13 performs recognition processing of traffic rules around the own vehicle based on the generated map information and data from each component of the vehicle control system 1 such as the position estimation unit 12 and the information detection unit 11. Performs processing such as recognition of vehicle-related situations. The traffic rule recognition process recognizes, for example, information such as the position and status of traffic signals around the host vehicle, details of traffic regulations around the host vehicle, and possible driving lanes.

For example, the situation analysis unit 13 may generate a local map for use in recognizing the situation around the own vehicle as the situation recognition map information. The situation of the vehicle to be recognized may include, for example, the vehicle's position, posture, movement (i.e., speed, acceleration, direction of movement, etc.), presence or absence of an abnormality, and if an abnormality occurs, its details. . The conditions surrounding the own vehicle that are subject to recognition include, for example, the types and positions of surrounding stationary objects, the types, positions, and movements of surrounding moving objects (e.g., speed, acceleration, direction of movement, etc.), and the surrounding roads. This may include configuration and road surface conditions, as well as surrounding weather, temperature, humidity, and brightness. The driver's condition to be recognized may include, for example, physical condition, alertness level, concentration level, fatigue level, line of sight movement, driving operation, and the like.
The situation analysis unit 13 outputs data indicating the recognition result to the position estimation unit 12. The situation analysis unit 13 may include the generated situation recognition map information in the data output to the position estimation unit 12. The situation analysis unit 13 stores the generated situation recognition map information in the storage unit 111.
Note that the situation analysis section 13 may output data indicating the recognition result to the information detection section 11. The information detection unit 11 may use data input from the situation analysis unit 13, for example, in order to detect information external to the host vehicle.
Moreover, information on the positions and areas of known roads, facilities, topography, etc. may be added to the above map information, situation recognition map information, and self-position estimation map.

The situation analysis unit 13 predicts the situation regarding the own vehicle based on generated map information, information on recognized traffic rules, information indicating the situation regarding the own vehicle, and other data from each component of the vehicle control system 1. Processing may be performed. For example, the situation analysis unit 13 performs predictive processing on the situation of the own vehicle, the situation around the own vehicle, the situation of the driver, and the like. The situation of the own vehicle to be predicted may include, for example, the behavior of the own vehicle, the occurrence of an abnormality, and the possible travel distance. The situation around the host vehicle that is to be predicted includes, for example, the behavior of moving objects around the host vehicle, changes in the state of traffic lights, and changes in the environment such as the weather. The driver's situation to be predicted may include, for example, the driver's behavior and physical condition.
The situation analysis section 13 outputs data indicating the processing result of the prediction process to the motion planning section 14 together with data indicating information on the recognized traffic rules and information on the situation regarding the host vehicle. Note that the situation analysis unit 13 may output data indicating the processing result of the prediction process to the information detection unit 11.

The motion planning section 14 plans a route to the destination based on data from each component of the vehicle control system 1, such as the situation analysis section 13. The motion planning unit 14 uses, for example, map information to determine a route from the current location to the designated destination. Further, for example, the motion planning unit 14 may change the determined route using conditions such as traffic jams, accidents, traffic regulations, construction, and the driver's physical condition.
The motion planning unit 14 plans the behavior of the own vehicle in order to safely travel within a planned time on a predetermined route based on data from each component of the vehicle control system 1, such as the situation analysis unit 13. . The motion planning unit 14 plans, for example, starting, stopping, traveling direction (for example, forward, backward, left turn, right turn, direction change, etc.), driving lane, driving speed, and overtaking.
The motion planning section 14 plans the motion of the host vehicle to realize a predetermined action based on data from each component of the vehicle control system 1, such as the situation analysis section 13. The motion planning unit 14 plans, for example, acceleration, deceleration, and travel trajectory. The motion planning section 14 outputs data indicating the planned motion of the own vehicle to the motion control section 15.

The operation control unit 15 performs detection processing for emergency conditions such as collision, contact, entry into a dangerous area, driver abnormality, vehicle abnormality, etc. based on data indicating detection results input from the information detection unit 11. . When detecting an emergency state, the operation control unit 15 plans an avoidance operation that is an operation of the own vehicle to avoid abnormal operations such as a sudden stop or a sudden turn.
The motion control section 15 performs acceleration/deceleration control to realize the motion of the host vehicle planned by the motion planning section 14. For example, the operation control unit 15 calculates a control target value of a driving force generating device or a braking device to realize planned acceleration and deceleration, and sends a control command indicating the calculated control target value to the drive system control unit 107. Output.

The motion control section 15 performs directional control to realize the motion of the host vehicle planned by the motion planning section 14. The operation control unit 15 calculates, for example, a control target value of the steering mechanism for realizing the planned travel trajectory, and outputs a control command indicating the calculated control target value to the drive system control unit 107.
In addition, when an avoidance operation is planned by the own unit, priority is given to, for example, a sharp turn as direction control for realizing the avoidance operation than the operation of the own vehicle planned by the movement planning unit 14. In that case, priority is given to, for example, a sharp turn as acceleration/deceleration control for realizing an avoidance action over the action of the own vehicle planned by the action planning unit 14.

Next, a configuration example of the information detection section 11 according to the present embodiment will be described. FIG. 2 is a schematic block diagram showing an example of the functional configuration of the information detection section 11 according to the present embodiment.
The information detection section 11 includes an object detection section 112, an object determination section 114, and an object detection control section 116. The object detection section 112 includes an object recognition section 112a, an area recognition section 112b, and a depth information processing section 112c. The information collection unit 102 includes a camera 102a and a distance measurement sensor 102b.

Image data is input to the object recognition unit 112a from the camera 102a. The camera 102a is installed in the host vehicle with its optical axis pointing in a predetermined direction (for example, forward). The camera 102a captures an image that appears within its field of view, captures the captured image using an image sensor, and generates image data that includes signal values for each pixel two-dimensionally arranged at regular intervals on the image sensor. . The position of each pixel included in one frame of image corresponds to a direction within the field of view of camera 102a. The camera 102a outputs the generated image data to the information collection unit 102. The camera 102a is, for example, a digital video camera that takes images at regular intervals (eg, 1/15 to 1/60 seconds).

The object recognition unit 112a performs image recognition processing on input image data, and detects an area where an object is represented in the image data as a first area (object detection). The object recognition unit 112a determines the type of object represented in the first area, and determines the reliability of the determined object type as the first reliability. The object recognition unit 112a uses, for example, a predetermined first machine learning model (object recognition model), inputs a signal value for each pixel indicated by image data as an input value, and estimates the signal value for each pixel as an output value. Calculate the reliability for each type of object.

The model parameters of the first machine learning model include, for each type of object, image data that indicates, as input values, signal values for each pixel representing an image in which the type and the area in which the object is represented are known, and the object. Contains multiple sets of output data (typically 1,000 to 10,000 or more) in which the output value for each pixel in the area where is expressed is 1, and the output value for each pixel in the other area is 0. It is obtained in advance by performing model learning (also called training) using training data. In model learning, model parameters are sequentially applied until convergence so that the difference between the output value and the calculated value for each pixel calculated using the first machine learning model for the input value approximates zero. Update. When the amount of change in the model parameter before and after the update or the amount of change in the magnitude of the difference before and after the update is less than a predetermined convergence determination threshold, it can be determined that the model parameter has converged.

As an index value of the magnitude of the difference, for example, an error function such as a sum of squared difference (SSD) or a cross entropy error can be used.
Therefore, when image data is input to the object recognition unit 112a, it is expected that the area in which the object appears and the type of the object will be specified as output. The calculated reliability indicates the accuracy with which the region and type of object appearing are identified.

The object recognition unit 112a determines, as a first region, a spatially continuous region in which the average value of reliability between pixels is equal to or higher than a predetermined reliability detection threshold and has the maximum value among object types. can be detected. The object recognition unit 112a determines the maximum value of the reliability as the first reliability, and determines the type of object that gives the maximum value as the first type. The object recognition unit 112a outputs object detection information indicating a set of the first reliability, the first region, and the first type to the object determination unit 114.

The area recognition unit 112b performs area division processing on the image data input from the camera 102a, divides the image data into areas each representing a different type of object, and divides each divided area into areas representing different types of objects. Determine the type of object (segmentation).
The area recognition unit 112b uses, for example, a predetermined second machine learning model (area recognition model), inputs the signal value for each pixel shown in the image data as an input value, and estimates the signal value for each pixel as an output value. Calculate the reliability of each type of object and identify the type of object that provides the highest reliability.

The model parameters of the second machine learning model are image data that indicates, as input values, signal values for each pixel representing an image in which a region in which multiple objects are represented and the types of individual objects are known; For each region represented, for each pixel in that region, training with multiple sets of output data in which the output value related to the object type is 1 and the output value related to other object types is 0. Obtain it in advance by performing model learning using the data. In model learning, the difference between the calculated value for each pixel and the output value calculated using the second machine learning model for the input value approximates zero for each object, so that Update model parameters.
Therefore, when image data is input to the region recognition unit 112b, it is expected that the image will be divided into regions in which individual objects appear as output, and the type of object in each region will be specified. The reliability indicates the accuracy with which an object appears in each region and the type of the object is identified. Note that in the second machine learning model, the object type may be set so that no object type is specified (unspecified). In that case, the area recognition unit 112b may define an area where the type of object is unspecified.

For example, the area recognition unit 112b selects a second area, which is a spatially continuous area, where the average value of reliability between pixels is equal to or higher than a predetermined reliability detection threshold and has the maximum value among object types. It can be detected as a region. The area recognition unit 112b determines the maximum value of the reliability as the second reliability, and determines the type of object that gives the maximum value as the second type. The area recognition unit 112b outputs area recognition information indicating a set of the second reliability, the second area, and the second type to the object determination unit 114. The area recognition unit 112b may include information indicating the type of object and the reliability of each pixel in the area recognition information, and output the information to the object determination unit 114.

Distance measurement data is input to the depth information processing section 112c from the distance measurement sensor 102b included in the information collection section 102. The distance measurement sensor 102b includes a radiation section that emits radiation waves, and a reception section that receives reflected waves reflected from the object surface. The distance sensor 102b can measure the distance to an object present in the direction of arrival of the reflected wave based on the phase difference between the radiated wave and the reflected wave and the traveling speed of the radiated wave. The range of the direction of arrival in which the distance measurement sensor 102b can receive reflected waves includes part or all of the field of view of the camera 102a. The ranging sensor 102b may be, for example, a LiDAR (Light Detection Ranging) system, a radar, or the like. LiDAR uses a highly convergent laser beam as a radiation wave. Radar uses electromagnetic waves (for example, millimeter waves) having a shorter wavelength than visible light or infrared rays as radiation waves.

The depth information processing unit 112c takes the arrival direction corresponding to the pixel of the image data acquired by the camera 102a as a sample point, and projects the distance measured for each sample point in the direction of the optical axis of the camera 102a. The resulting cosine component is calculated as the depth to the object. The depth information processing unit 112c outputs depth information indicating the depth calculated for each sample point to the object determination unit 114.

Note that the depth information processing unit 112c performs region division processing on the distance measurement data input from the distance measurement sensor 102b, dividing it into regions each representing a different type of object, and for each divided region, The reliability of the area may be determined. The depth information processing unit 112c does not necessarily need to determine the type of object as long as it can determine the area in which each object appears based on the depth.
The depth information processing unit 112c uses, for example, a predetermined third machine learning model (region recognition model), inputs a signal value for each sample point corresponding to a pixel as an input value, and estimates the signal value for each pixel as an output value. The reliability may be calculated for each region of the object.

The model parameters of the third machine learning model are image data that indicates, as input values, signal values for each pixel representing an area in which multiple objects are represented, and signal values for each pixel in the area in which individual objects are represented. , is obtained in advance by performing model learning using training data comprising a plurality of sets of output data in which the output value for that region is 1 and the output value for other regions is 0. In model learning, the difference between the calculated value for each pixel and the output value calculated using the third machine learning model for the input value approximates zero for each region, until convergence is achieved. Update model parameters.

Therefore, when image data is input to the depth information processing section 112c, it is expected that the image will be divided into regions in which individual objects appear as output. Reliability indicates the accuracy with which individual regions are identified. Note that the third machine learning model may be set to not belong to any area (unspecified). In that case, the depth information processing unit 112c may define an unspecified area.
The depth information processing unit 112c individually selects a spatially continuous area in which the average value between pixels of the reliability is equal to or higher than a predetermined reliability detection threshold and has the maximum value individually as the area where the object appears. It can be detected as three areas. The area recognition unit 112b may add information indicating the third area to the depth information and output it to the object determination unit 114.

Note that the first machine learning model, the second machine learning model, and the third machine learning model include, for example, neural networks such as convolutional neural networks and recurrent neural networks; Alternatively, models other than neural networks such as decision trees and support vector machines can be applied. The object detection unit 112 includes a model learning unit that performs learning using training data to generate model parameters for each of the first machine learning model, the second machine learning model, and the third machine learning model. You may prepare.

Object recognition information, area recognition information, and depth information are input to the object determination unit 114 from the object detection unit 112.
When the first reliability indicated by the object recognition information is less than a predetermined detection threshold, the object determination unit 114 rejects the first reliability and the first region and first type corresponding to the first reliability. Then, a first reliability level that is equal to or higher than a predetermined detection threshold value, and a first area and a first type corresponding to the first reliability level are stored.
When the second reliability indicated by the region recognition information is less than a predetermined detection threshold, the object determination unit 114 rejects the second reliability and the second region and second type corresponding to the second reliability. Then, a second reliability level that is equal to or higher than a predetermined detection threshold value, and a second area and a second type corresponding to the second reliability level are stored. Therefore, there may be areas in the image to be processed where no objects are detected, or there may be cases where no objects are detected.

The object determination unit 114 adjusts the second region indicated by the region recognition information based on the depth information so that the depth of the object represented in the second region is maintained within a predetermined range. do. For example, the object determination unit 114 determines that the average value of the adjusted depth of the second area is within a predetermined range based on the average value of the depth of each pixel included in the second area, and that the second area is the maximum value. Expanding or contracting a portion of the second area so that the second area is
When the depth information includes information indicating a third region, the object determination unit 114 identifies a third region that partially or completely overlaps with the second region, and identifies pixels in the third region. The depth distribution range for each pixel in the second area after adjustment falls within the depth distribution range for each pixel in the second area, and a part of the second area is Expand or contract.

The object determining unit 114 determines that the type of object detected in a non-overlapping area that does not overlap with the first area in the second area is the second type. Further, the object determining unit 114 may determine that the type of the detected object is the first type in a non-overlapping area that does not overlap with the second area in the first area.
The object determining unit 114 determines that the object type is the second type when the first reliability is less than a determination threshold that is a predetermined reliability threshold in an overlapping region where the first region and the second region overlap. The object determining unit 114 determines that the type of object in the overlapping area is the first type when the first reliability is equal to or higher than the discrimination threshold in the overlapping area. The discrimination threshold may be equal to the detection threshold or may be higher than the detection threshold. By using a value higher than the detection threshold as the discrimination threshold, it is possible to emphasize the recognition results obtained by area recognition.
The object determination unit 114 outputs object determination information indicating the type of the determined object and its area in the image to the position estimation unit 12, the situation analysis unit 13, and the operation control unit 15. A region in an image is represented by the position occupied by the object, that is, the range of direction for each pixel and the range of depth for each pixel.

The object detection control unit 116 determines whether to operate the area recognition unit 112b and the depth information processing unit 112c based on the situation of the own vehicle and the surroundings of the own vehicle. The object detection control unit 116 receives data indicating the own vehicle and the surrounding situation of the own vehicle inputted from the situation analysis unit 13, and the input data is used for determination. The input data may include part of the data input from various sensors forming the information collection unit 102. The data indicating the surrounding situation of the host vehicle may include information regarding the object detected by the object detection unit 112, such as the position, speed, etc.

All or some of the following operating conditions are set in advance as determination targets in the object detection control unit 116, and it is determined whether each of the operating conditions to be determined is satisfied. The following operating conditions also include conditions regarding objects or areas detected by the object recognition unit 112a. When any of the operating conditions is satisfied, the object detection control unit 116 determines to operate the area recognition unit 112b and the depth information processing unit 112c, and when any of the operating conditions are not satisfied, the object detection control unit 116 It is determined that the recognition unit 112b and depth information processing unit 112c are not operated. When determining to operate the area recognition unit 112b and depth information processing unit 112c, the object detection control unit 116 provides the object detection unit 112 with control information for instructing the operation of the area recognition unit 112b and depth information processing unit 112c. Output.
When determining that the area recognition unit 112b and the depth information processing unit 112c are not to be operated, the object detection control unit 116 transmits control information to the object detection unit 112 for instructing the area recognition unit 112b and the depth information processing unit 112c to stop operating. Output to. The area recognition unit 112b and the depth information processing unit 112c each execute their own operation when control information for instructing an operation is input from the object detection control unit 116. The area recognition unit 112b and the depth information processing unit 112c stop their operations if they are currently executing their own operations when control information for instructing them to stop their operations is input from the object detection control unit 116. do.

Any one of the following conditions or a predetermined combination can be used as the operating conditions for the area recognition section 112b and the depth information processing section 112c.
(1) The existence of an object whose distance from the device itself is equal to or greater than a predetermined distance threshold is estimated. For example, the object detection control unit 116 may directly determine whether or not there is an object whose distance from the own device, that is, the own vehicle, is a predetermined distance threshold. The object detection control unit 116 detects an event that is likely to cause an object whose distance from the own device is equal to or greater than a predetermined distance threshold, such as the fact that the own vehicle is within an expressway area. It may be determined whether or not this has occurred. The object detection control unit 116 can, for example, refer to the map information and determine whether the position of the own vehicle is within the expressway area indicated by the map information. In addition, when detecting radio waves emitted from ETC (Electronic Toll Collection) systems installed at entrances and exits to expressways with an electric field intensity equal to or higher than a predetermined electric field intensity threshold, the object detection control unit 116 performs a high-speed It may be determined that the vehicle has entered a road or exited from an expressway. During the period from when the vehicle enters the expressway to when it leaves the expressway, the object detection control unit 116 can determine that the own vehicle is within the area of the expressway.

(2) The situation around the device indicates a predetermined situation in which it is difficult to detect the first area from the image data. For example, the object detection control unit 116 determines whether the illuminance indicated by the illuminance information input directly or indirectly from a light receiving sensor included in the vehicle is equal to or higher than a predetermined illuminance threshold. The object detection control unit 116 may determine whether fog information input from a fog sensor included in the vehicle indicates detection of fog. The object detection control unit 116 may determine whether rainfall information input from a raindrop sensor included in the vehicle indicates rain. The object detection control unit 116 may determine whether an input signal indicating a wiper operation forming the operation unit 101 is input.

(3) The number of objects around the host vehicle, that is, the number of objects detected by the object detection unit 112 is equal to or greater than a predetermined number threshold, or the spatial The interval is equal to or greater than a predetermined interval threshold. The spatial interval may be an interval in an actual three-dimensional space, or an interval expressed in terms of the number of pixels for a plurality of objects detected from an image taken by the camera 102a.
(4) The type of object detected by the object recognition unit 112a is a predetermined type that tends to cause misrecognition, or no type of object is determined, that is, the type of the object is unknown. thing.

(5) When the speed of the host vehicle exceeds a predetermined speed threshold, or the collision prediction time based on the distance from the host vehicle to the object detected by the object recognition unit 112a and the speed of the host vehicle is a predetermined collision prediction. be below the time threshold.
The predicted collision time is calculated by dividing the difference between the speed of the own vehicle and the speed of the detected object by the distance to the detected object. The velocity of an object can be calculated by differentiating the position of the object with respect to time.

Next, an example of the object detecting section 112 and the object determining section 114 according to the present embodiment will be described using the case where image data shown in FIG. 3 is used as an example. FIG. 3 exemplifies a case where a vehicle crosses a road extending in the depth direction from right to left with respect to the drawing at an intersection.
FIG. 4 illustrates a first area representing an object detected by the object recognition unit 112a. In this example, the area where the vehicle is displayed is detected as the first area R01.
FIG. 5 illustrates the regions divided by the region recognition unit 112b and the second region in which the object is represented. In FIG. 5, each display pattern (hatching) indicates a divided area. Among them, a region whose reliability is equal to or higher than a predetermined detection threshold is detected as a second region R02'. Note that R02 indicates the second region after adjustment.
FIG. 6 shows a depth image showing the depth of each sample point acquired by the depth information processing unit 112c. Each sample point corresponds to a pixel. Depth is indicated by shading; the darker the area, the lower (closer) the depth, and the brighter the area, the higher (further) the depth.

In the examples shown in FIGS. 4 to 6, the shapes of the first region R01, the second region R02', and the adjusted second region R02 are each normalized to a rectangle. As a result of the normalization, the first region R01 and the second region R02' include regions whose reliability is less than or equal to a predetermined detection threshold, but by normalizing the shape, the region where the object exists is simply defined.
For example, the object recognition unit 112a detects the vertical direction (Y direction) coordinate value of the pixel between the minimum value and the maximum value of the horizontal direction (X direction) coordinate value of the pixel whose reliability is equal to or higher than a predetermined detection threshold. The area between the minimum value and the maximum value is defined as the first region R01. The object recognition unit 112a can determine the average value of reliability between pixels in the first region R01 as the first reliability.

The region recognition unit 112b, for example, determines, as the second region R02', a region in which a predetermined type of object to be detected (in the example of FIG. 5, a vehicle) appears among the divided individual regions. More specifically, the area recognition unit 112b detects a pixel between the minimum value and the maximum coordinate value in the horizontal direction (X direction) of the pixel for which the reliability related to the object of the type is equal to or higher than a predetermined detection threshold. The area between the minimum and maximum coordinate values in the vertical direction (Y direction) can be defined as the second region R02'. The region recognition unit 112b can determine the average value of reliability between pixels in the second region R02' as the second reliability. Note that normalizing the shape may cause overlap between a plurality of regions, but the overlap of regions may be allowed.

The object determination unit 114 analyzes the second region R02′ defined by the region recognition unit 112b for each sample point corresponding to each pixel in the second region R02′ based on the depth information acquired by the depth information processing unit 112c. The second region R02' is adjusted so that the depth of the second region R02' is maintained within a predetermined range, and the adjusted second region R02 is defined. The object determination unit 114 determines, for example, that the magnitude of the difference in depth between any pixel at the left end of the second region R02′ and the pixel adjacent to the left (in the opposite horizontal direction) is a predetermined value. When there is a pixel whose difference is equal to or less than the threshold, the left end of the second region R02' is extended to the left adjacent pixel. Then, the process of expanding the left end of the second region R02' is repeated until there is no pixel for which the size of the difference in depth between the pixel adjacent to the left is less than or equal to a predetermined difference threshold. The object determining unit 114 can also expand the second region R02' to the right (horizontal direction) using the same method as for expanding to the left. The object determination unit 114 can also expand the second region R02' upward (in the vertical direction) and downward (in the opposite direction to the vertical direction) using the same method as in the leftward expansion.

The object determining unit 114 determines that the type of object detected in a non-overlapping area that does not overlap with the first area R01 in the second area R02 is a second type.
The object determination unit 114 determines the type of object as a second type when the first reliability is less than a discrimination threshold that is a predetermined reliability threshold in an overlapping region that overlaps the first region R01 and the second region R02. do. The information processing device 10 determines that the type of object in the overlapping area is the first type when the first reliability is equal to or higher than the discrimination threshold in the overlapping area.
Note that the object determining unit 114 may determine that the type of object detected in a non-overlapping area that does not overlap with the second area R02 in the first area R01 is the first type.

Next, an example of object detection processing according to this embodiment will be described. FIG. 7 is a flowchart illustrating an example of object detection processing according to this embodiment.
(Step S102) The camera 102a photographs an image within its field of view around itself, and outputs image data representing the photographed image to the information detection unit 11. Thereafter, the process advances to step S104.
(Step S104) The object recognition unit 112a detects an area in which an object is represented from the image indicated by the image data input from the camera 102a as a first area, and determines the type of object represented in the first area as a first area. It is determined as a type, and its reliability is determined as the first reliability. Thereafter, the process advances to step S106.

(Step S106) The area recognition unit 112b divides the image indicated by the image data into areas in which different types of objects are represented as second areas, and divides the image indicated by the image data into areas in which objects of different types are represented, and divides each second area into areas where objects of different types are represented. The type is defined as the second type. Thereafter, the process advances to step S108.
(Step S108) The depth information processing unit 112c acquires depth information indicating the depth of the object for each sample point corresponding to each pixel from the distance information input from the distance measurement sensor 102b. Thereafter, the process proceeds to step S110.

(Step S110) The object determination unit 114 adjusts the second region based on the depth information so that the depth of the object represented in the second region is maintained within a predetermined range. Thereafter, the process advances to step S112.
(Step S112) The object determination unit 114 determines whether the second area overlaps with the first area. When it is determined that there is overlap (step S112 YES), the process advances to step S114. When it is determined that there is no overlap (step S112 NO), the process advances to step S118.
(Step S114) The object determination unit 114 determines whether the first reliability related to the first region is greater than or equal to a predetermined determination threshold. When it is determined that it is equal to or greater than the determination threshold (step S114 YES), the process proceeds to step S116. When it is determined that it is less than the determination threshold (step S114 NO), the process advances to step S118.

(Step S116) The object determination unit 114 determines the type of object represented in the overlapping area as the first type. Thereafter, the process of FIG. 7 is ended.
(Step S118) The object determination unit 114 determines the type of object represented in the overlapping area or the second area as the second type. Thereafter, the process of FIG. 7 is ended.

Next, an example of object detection control according to this embodiment will be described. FIG. 8 is a state transition diagram showing an example of object detection control according to this embodiment. However, it is assumed that the state of step S202 is adopted when power is turned on to the information processing device 10, that is, at the beginning of power supply.
(Step S202) In the object detection unit 112, the object recognition unit 112a operates, and the area recognition unit 112b and depth information processing unit 112c do not operate. The object detection control unit 116 determines whether each preset operating condition to be determined is satisfied. When the object detection control unit 116 determines that any of the operating conditions is satisfied, the object detection control unit 116 activates the area recognition unit 112b and the depth information processing unit 112c, and proceeds to step S204.
(Step S204) In the object detection unit 112, the object recognition unit 112a, the area recognition unit 112b, and the depth information processing unit 112c operate, and all are used in combination.
When the object detection control unit 116 determines that none of the preset operating conditions to be determined are satisfied, the object detection unit 112 returns to the state of step S202.

In the above description, the object detection control unit 116 controls whether or not to operate the area recognition unit 112b and the depth information processing unit 112c immediately after determining the operating conditions. Not limited.
When a state that satisfies any of the operation conditions to be determined continues for a predetermined period of time (for example, 30 seconds to 3 minutes), the object detection control unit 116 operates the area recognition unit 112b and the depth information processing unit 112c. You may judge. Furthermore, when a state in which the operating conditions to be determined are not satisfied continues for a predetermined period of time or more, the object detection control unit 116 determines that the area recognition unit 112b and the depth information processing unit 112c are not operated. It's okay.

(minimum configuration)
Next, the minimum configuration of this embodiment will be explained. FIG. 9 is a schematic block diagram illustrating the minimum configuration of this embodiment.
As shown in FIG. 9, the information processing device 10 according to the present embodiment includes an object detection unit 112 that detects objects around the device, and an object determination unit 114 that determines the type of the detected object. The object detection section 112 includes an object recognition section 112a and an area recognition section 112b. The object recognition unit 112a detects a first region, which is a region where an object is represented in the image, from image data representing an image using signal values of each of a plurality of pixels, and determines the type of object represented in the first region. A first reliability level, which is a first type of reliability level, is determined. The area recognition unit 112b divides the image into second areas representing different types of objects based on the image data, and divides each second area into a second area representing the type of object represented in the second area. Define the type. The object determining unit 114 determines the type of object in a non-overlapping area that does not overlap with the first area in the second area as the second type, and determines the type of object in the overlapping area that overlaps with the first area in the second area as the first type. If the reliability is less than a predetermined reliability threshold, the type of object in the overlapping region is determined as a second type, and if the first reliability is greater than or equal to the predetermined reliability threshold in the overlapping region, The type of object in the overlapping area is determined as the first type.
According to this configuration, object recognition and area recognition for image data are used together. According to area recognition, the image is divided into areas for different objects represented in the image, so objects can be detected more reliably than object recognition, which specifies the area in which the object is represented. Further, when the reliability of the region in which the object is represented by object recognition is high, the type of object determined by object recognition is obtained as a determination result. Therefore, it is possible to detect objects and determine their types more reliably than when using object recognition alone.

Further, the object detection unit 112 includes a depth information processing unit 112c that acquires depth information indicating the depth of the object for each sample point corresponding to a pixel, and the object determination unit 114 includes a depth information processing unit 112c that acquires depth information indicating the depth of the object for each sample point corresponding to a pixel. The second region may be adjusted such that the second region is maintained within a predetermined range.
According to this configuration, the second area is adjusted so that the depth falls within a predetermined range, so areas with significantly different depths are excluded from the second area, and areas with a common depth are included in the second area. It will be done. In addition, depth can be measured more stably than images even in low illuminance or bad weather. Therefore, it is possible to more accurately estimate the area in which each object is represented as the second area. For example, even when objects of the same or similar type are adjacent, individual regions are divided based on depth.

Further, when the information processing device 10 estimates the presence of an object whose distance from the device is equal to or greater than a predetermined distance threshold among the objects detected by the object recognition unit 112a, the information processing device 10 uses the area recognition unit 112b and the depth information processing unit 112c. The object detection control section 116 may be provided to operate the object detection control section 116.
According to this configuration, by operating the area recognition unit 112b and the depth information processing unit 112c for objects that are far from the own device, it is possible to more reliably detect the area and type of object displayed. . When a distant object is not detected, the area recognition unit 112b and the depth information processing unit 112c are allowed to stop operating, so the amount of processing can be reduced.

In addition, when the information processing device 10 indicates a predetermined environment (for example, rain, dense fog, dark area) in which it is difficult to detect the first region from the image data, the information processing device 10 uses the region recognition unit 112b and the depth An object detection control section 116 that operates the information processing section 112c may be provided.
According to this configuration, when it becomes difficult to detect the first area in which the object is represented by object recognition, the object is displayed more reliably by operating the area recognition unit 112b and the depth information processing unit 112c. It is possible to detect the area and type of If the detection of the first region does not become difficult, the operation of the region recognition section 112b and the depth information processing section 112c is allowed to stop, so that the amount of processing can be reduced.

In addition, the information processing device 10 determines when the number of objects detected by the object recognition unit 112a is equal to or greater than a predetermined number threshold, or when the interval between the first regions of each object detected by the object recognition unit 112a is a predetermined number. An object detection control unit 116 may be provided that operates the area recognition unit 112b and the depth information processing unit 112c when the interval is equal to or less than a threshold value.
According to this configuration, when a plurality of objects are detected at once or when the areas of detected objects are close to each other, the area recognition unit 112b and the depth information processing unit 112c are operated to improve accuracy. It is possible to detect the area and type in which objects appear. When the number of detected objects is small and they are not close to each other, the area recognition section 112b and the depth information processing section 112c are allowed to stop operating, so that the amount of processing can be reduced.

Furthermore, when the type of object detected by the object recognition unit 112a is a predetermined type, or when the type of the object is not determined by the object recognition unit 112a, the information processing device 10 performs depth information processing with the area recognition unit 112b. The object detection control section 116 may be provided to operate the section 112c.
According to this configuration, when the type of a predetermined object tends to be misrecognized or when the type of the object cannot be specified, the area recognition unit 112b and the depth information processing unit 112c are operated to improve the accuracy of the recognition process. The type of object can be detected reliably. If the type of object is specified and the detected object is an object that does not tend to be misrecognized, stopping the operation of the area recognition unit 112b and depth information processing unit 112c is allowed, so that the amount of processing can be reduced. can be reduced.

In addition, when the speed of the vehicle in which the information processing device 10 is mounted becomes equal to or higher than a predetermined speed threshold, or when the information processing device 10 detects a collision based on the distance from the own portion to the object detected by the object recognition unit 112a and the speed of the vehicle, the information processing device 10 The object detection control unit 116 may be provided to operate the area recognition unit 112b and the depth information processing unit 112c when the predicted time is equal to or less than a predetermined collision prediction time threshold.
According to this configuration, when the traveling speed increases or a collision with an object is expected, by operating the area recognition unit 112b and the depth information processing unit 112c, the area where the object is more reliably represented. can be detected. Therefore, it is possible to contribute to realizing more accurate control in situations where safety is required. When the traveling speed is low and a collision with an object is expected, the area recognition section 112b and the depth information processing section 112c are allowed to stop operating, so the amount of processing can be reduced.

Note that the information processing device 10 described above may include a computer system therein. For example, the processor described above can be a component of a computer system. The ECU also includes a computer system. Each of the above-mentioned processing steps is stored in a computer-readable storage medium in the form of a program, and the computer reads and executes this program to perform the processing. The computer system includes software such as an OS (Operation System), device drivers, and utility programs, and hardware such as peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as magnetic disks, magneto-optical disks, ROMs (Read Only Memory), and semiconductor memories, and storage devices such as hard disks built into computer systems. Furthermore, a computer-readable recording medium refers to a medium that dynamically stores a program for a short period of time, such as a communication line used to transmit a program using a network such as the Internet or a communication line such as a telephone line. It may also include a device that retains a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client. Furthermore, the above-mentioned program may be one for realizing a part of the above-mentioned functions.Furthermore, the above-mentioned program may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system, a so-called difference file (difference file). program).

Furthermore, part or all of the information processing device 10 in the embodiment described above may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the information processing device 10 may be made into a processor individually, or some or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, but may be implemented using a dedicated circuit or a general-purpose processor. Further, if an integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Additions, omissions, substitutions, and other changes to the configuration are possible without departing from the spirit of the invention.
Moreover, the invention is not limited by the foregoing description, but only by the scope of the appended claims.

According to the information processing apparatus, information processing method, and program of each of the above aspects, object recognition and area recognition for image data are used together. According to area recognition, the image is divided into areas for different objects represented in the image, so objects can be detected more reliably than object recognition, which specifies the area in which the object is represented. Further, when the reliability of the region in which the object is represented by object recognition is high, the type determined by object recognition is obtained as a determination result. Therefore, it is possible to detect objects and determine their types more reliably than when using object recognition alone.

DESCRIPTION OF SYMBOLS 1... Vehicle control system, 10... Information processing device, 11... Information detection part, 12... Position estimation part, 13... Situation analysis part, 14... Motion planning part, 15... Motion control part, 101... Operation part, 102... Information Collection unit, 102a...Camera, 102b...Distance sensor, 103...Communication unit, 104...In-vehicle equipment, 105...Output control unit, 106...Output unit, 107...Drive system control unit, 108...Drive system, 109...Vehicle system Control unit, 110... Vehicle system, 111... Storage unit, 112... Object detection unit, 112a... Object recognition unit, 112b... Area recognition unit, 112c... Depth information processing unit, 114... Object determination unit, 116... Object detection control unit

Claims (8)

an object detection unit that detects objects surrounding the device; and an object determination unit that determines the type of the object,
The object detection section includes an object recognition section, an area recognition section, and a depth information processing section,
The object recognition unit includes:
A first area, which is an area in which an object is represented in the image, is detected from image data representing an image using signal values of each of a plurality of pixels, and a first area, which is the type of object represented in the first area, is detected. Determine the first reliability level, which is the reliability level of the type,
The depth information processing section includes:
Obtaining depth information indicating the depth of the object for each sample point corresponding to the pixel,
The area recognition unit is
dividing the image into second regions representing different types of objects from the image data, and determining a second type representing the type of object represented in the second region for each second region;
The object determination unit includes:
adjusting the second area so that the depth of the object represented in the second area is maintained within a predetermined range;
Determining the type of object in a non-overlapping area that does not overlap with the first area in the second area as a second type;
If the first reliability is less than a predetermined reliability threshold in an overlapping region that overlaps with the first region in the second region,
Determining the type of object in the overlapping area as a second type,
In the overlapping region, when the first reliability is greater than or equal to a predetermined reliability threshold,
An information processing device that determines the type of object in the overlapping area as a first type. an object detection control unit that operates the area recognition unit and the depth information processing unit when the presence of an object whose distance from the device is greater than or equal to a predetermined distance threshold among the objects detected by the object recognition unit; The information processing device according to claim 1 . The device further comprises an object detection control unit that operates the area recognition unit and the depth information processing unit when environmental information around the device indicates a predetermined environment in which it is difficult to detect the first area from the image data. The information processing device according to claim 1 or claim 2 . When the number of objects detected by the object recognition unit is equal to or greater than a predetermined number threshold, or when the interval between the first regions of each of the objects is equal to or less than a predetermined interval threshold, the area recognition unit and the The information processing device according to any one of claims 1 to 3 , further comprising an object detection control unit that operates a depth information processing unit. Claim: an object detection control unit that operates the area recognition unit and the depth information processing unit when the type of object detected by the object recognition unit is a predetermined type, or when the type of the object is not determined. The information processing device according to any one of claims 1 to 4 . When the speed of the vehicle carrying the self-portion is equal to or higher than a predetermined speed threshold, or when the predicted collision time is based on the distance from the self-portion to the object detected by the object recognition section and the speed of the vehicle, a predetermined collision prediction occurs. The information processing device according to any one of claims 1 to 5 , further comprising an object detection control unit that operates the area recognition unit and the depth information processing unit when the time becomes equal to or less than a time threshold. An information processing method in an information processing device, the method comprising:
The information processing device
A first area, which is an area in which an object is represented in the image, is detected from image data representing an image using signal values of each of a plurality of pixels, and a first area, which is the type of object represented in the first area, is detected. a first step of determining a first reliability that is a reliability of a type;
a second step of acquiring depth information indicating the depth of the object for each sample point corresponding to the pixel;
dividing the image into second regions representing different types of objects from the image data, and determining a second type of object represented in the second region for each second region; 3 steps and
a fourth step of adjusting the second area so that the depth of the object represented in the second area is maintained within a predetermined range;
Determining the type of object in a non-overlapping area that does not overlap with the first area in the second area as a second type;
If the first reliability is less than a predetermined reliability threshold in an overlapping region that overlaps with the first region in the second region,
Determining the type of object in the overlapping area as a second type,
In the overlapping region, when the first reliability is greater than or equal to a predetermined reliability threshold,
a fifth step of determining the type of object in the overlapping area as a first type;
An information processing method that performs. to the computer,
an object detection unit that detects objects surrounding the device; and an object determination unit that determines the type of the object,
The object detection section includes an object recognition section, an area recognition section, and a depth information processing section,
The object recognition unit is
A first area, which is an area in which an object is represented in the image, is detected from image data representing an image using signal values of each of a plurality of pixels, and a first area, which is the type of object represented in the first area, is detected. Determine the first reliability level, which is the reliability level of the type,
The depth information processing section includes:
Obtaining depth information indicating the depth of the object for each sample point corresponding to the pixel,
The area recognition unit includes:
dividing the image into second regions representing different types of objects from the image data, and determining a second type representing the type of object represented in the second region for each second region;
The object determination unit includes:
adjusting the second area so that the depth of the object represented in the second area is maintained within a predetermined range;
Determining the type of object in a non-overlapping area that does not overlap with the first area in the second area as a second type;
If the first reliability is less than a predetermined reliability threshold in an overlapping region that overlaps with the first region in the second region,
Determining the type of object in the overlapping area as a second type,
In the overlapping region, when the first reliability is equal to or higher than a predetermined reliability threshold,
A program for operating as an information processing device that determines the type of object in the overlapping area as a first type.

Images