HK40068772A - Real-time sensor calibration and calibration verification based on statically mapped objects - Google Patents

Description

Real-time sensor calibration and calibration verification based on statically mapped objects

Cross Reference to Related Applications

This application claims the benefit of U.S. provisional application No.63/069,918 filed on 25/8/2020, 35u.s.c.119(e), the contents of which are incorporated herein in their entirety.

Technical Field

The present invention relates generally to sensor calibration and, more particularly, in some embodiments, to real-time calibration and calibration verification of sensors using static mapping objects present in an operating environment.

Background

On-board sensors in a vehicle, such as an autonomous vehicle, supplement and enhance the field of view (FOV) of the vehicle by providing a continuous stream of sensor data captured from the vehicle's surroundings. Sensor data is used in conjunction with various vehicle-based applications including, for example, blind spot detection, lane change assistance, rear-end radar for collision warning or avoidance, parking assistance, cross-traffic monitoring, brake assistance, emergency braking, and automatic distance control.

In-vehicle sensors include, for example, cameras, light detection and ranging (LiDAR) systems, radar-based systems, Global Positioning System (GPS) devices, sonar-based sensors, ultrasonic sensors, Inertial Measurement Units (IMU), accelerometers, gyroscopes, magnetometers, and Far Infrared (FIR) sensors. The sensor data may include image data, reflected laser data, and the like. Typically, images captured by onboard sensors utilize a three-dimensional (3D) coordinate system to determine distances and angles of objects in the images relative to each other and to the vehicle. In particular, such real-time spatial information is acquired in the vicinity of the vehicle using various on-board sensors distributed throughout the vehicle, and then the real-time spatial information may be processed to calculate various vehicle parameters and determine safe driving operation of the vehicle.

Autonomous vehicles perform a large number of complex calculations based on sensor data captured from on-board vehicle sensors to facilitate various operations required for autonomous vehicle operations, such as object detection, semantic object segmentation, object tracking, collision avoidance, vehicle navigation, vehicle acceleration and deceleration, and the like. To ensure that such operations are performed in compliance with the stringent safety standards required by autonomous vehicles, the calculations must have a high degree of precision, which in turn requires that the sensor data on which the calculations are based have a high level of precision and accuracy. The accuracy and precision of the sensor data, in turn, depends on whether the sensor that captured the sensor data was properly calibrated. Discussed herein are technical solutions to solve technical problems related to sensor calibration, in particular to calibration of sensors related to a vehicle such as an autonomous vehicle.

Disclosure of Invention

In an exemplary embodiment, a computer-implemented method for calibrating a first sensor associated with a vehicle using a static mapping object is disclosed. The computer-implemented method includes: capturing, via a first sensor, point cloud data, at least a portion of which is associated with an object in an environment through which a vehicle is traveling; and determining that the object is a statically mapped object. The computer-implemented method also includes determining that the statically mapped objects satisfy one or more criteria for calibrating the first sensor, and calibrating the first sensor to a global coordinate system via a second sensor associated with the vehicle relative to the point cloud data, wherein the second sensor has been pre-calibrated to the global coordinate system.

In an exemplary embodiment, determining that the object is a statically mapped object includes determining a location of the object within a global coordinate system, retrieving map data of an environment through which the vehicle travels, and determining that the location of the object corresponds to a location of the statically mapped object in the map data.

In an exemplary embodiment, calibrating the first sensor to the global coordinate system includes determining a transformation matrix from a first local coordinate system of the first sensor to a second local coordinate system of the second sensor.

In an exemplary embodiment, determining the transformation matrix comprises determining at least one rotation component and at least one translation component of the transformation matrix.

In an exemplary embodiment, the at least one rotational component and the at least one translational component align at least a portion of the point cloud data with a position of the static mapping object within the global coordinate system as a relative position of the vehicle with respect to the static mapping object changes.

In an exemplary embodiment, calibrating the first sensor includes determining whether the transformation matrix results in a calibration accuracy of the first sensor being within a calibration accuracy threshold.

In an exemplary embodiment, the transformation matrix is a first transformation matrix, and calibrating the first sensor to the global coordinate system further comprises: determining a location of the object in the first local coordinate system based at least in part on at least a portion of the point cloud data associated with the object; applying the first transformation matrix to a position of the object in the first local coordinate system to obtain a corresponding position in the second local coordinate system; applying the second transformation matrix to the corresponding location in the second local coordinate system to obtain a calibrated location of the object in the global coordinate system; and determining a calibration accuracy of the calibration of the first sensor based at least in part on a deviation between a calibrated location of the object in the global coordinate system obtained via applying the first and second transformation matrices and a location of the statically mapped object in the map data.

In an exemplary embodiment, determining whether the transformation matrix causes the calibration accuracy of the first sensor to be within the calibration accuracy threshold includes determining that the calibration accuracy of the first sensor is not within the calibration accuracy threshold, at least a portion of the point cloud data associated with the object is a first portion of the point cloud data, and the static mapping object is a first static mapping object. In an exemplary embodiment, the method further comprises determining that a second object in the environment through which the vehicle is traveling is a second statically mapped object, the second portion of the point cloud data being associated with the second statically mapped object; re-determining the transformation matrix relative to a second portion of the point cloud data; and re-determining the calibration accuracy of the first sensor based at least in part on the re-determined transformation matrix.

In an exemplary embodiment, determining that the static mapping object satisfies the one or more criteria for calibrating the first sensor includes determining at least one of a shape feature, a size feature, or a surface feature of the static mapping object; determining a similarity metric indicative of a level of similarity between at least one of a shape feature, a size feature, or a surface feature of the statically mapped object and one or more features preselected to be optimal for calibrating the first sensor; and determining that the similarity metric satisfies a threshold.

In an exemplary embodiment, the method further includes determining a calibration error associated with the first sensor after calibrating the first sensor, determining that the calibration error exceeds a threshold acceptable calibration error, and recalibrating the first sensor at least partially in response to determining that the calibration error exceeds the threshold acceptable calibration error.

In an exemplary embodiment, the method further comprises, prior to calibrating the first sensor, identifying data indicative of a set of environmental conditions in an environment through which the vehicle is traveling; determining that the set of environmental conditions meets one or more criteria for optimal calibration of the first sensor; and initiate calibration of the first sensor at least partially in response to determining that the set of environmental conditions satisfies one or more criteria for optimal calibration of the first sensor.

In an exemplary embodiment, the data indicative of the set of environmental conditions includes at least one of i) third party data indicative of one or more of weather conditions, time of day, or density of static objects in the portion of the environment, or ii) sensor data from at least a third sensor associated with the vehicle, wherein the third sensor includes one or more of a humidity sensor, a thermal sensor, or a vibration sensor.

In an exemplary embodiment, the first sensor is a LiDAR sensor and the second sensor is an inertial sensor, such as an IMU associated with a GPS device.

In an exemplary embodiment, a system for calibrating a first sensor associated with a vehicle using a static mapping object is disclosed. The system includes at least one processor and at least one memory storing computer-executable instructions. The at least one processor is configured to access the at least one memory and execute the computer-executable instructions to perform a set of operations comprising capturing, via the first sensor, point cloud data, at least a portion of which is associated with an object in an environment through which the vehicle is traveling; and determining that the object is a statically mapped object. The set of operations also includes determining that the statically mapped object satisfies one or more criteria for calibrating the first sensor, and calibrating the first sensor to a global coordinate system via a second sensor associated with the vehicle relative to the point cloud data, wherein the second sensor has been pre-calibrated to the global coordinate system.

The above-described system is further configured to perform any of the operations/functions and may include any of the additional features/aspects of the exemplary embodiments of the present invention described above with respect to the exemplary computer-implemented methods of the present invention.

In an exemplary embodiment, a computer program product for calibrating a first sensor associated with a vehicle using a static mapping object is disclosed. The computer program product includes a non-transitory computer readable medium readable by a processing circuit. A non-transitory computer readable medium stores instructions executable by a processing circuit to cause a method to be performed. The method comprises the following steps: capturing, via a first sensor, point cloud data, at least a portion of which is associated with an object in an environment through which a vehicle is traveling; and determining that the object is a statically mapped object. The computer-implemented method also includes determining that the statically mapped objects satisfy one or more criteria for calibrating the first sensor, and calibrating the first sensor to a global coordinate system via a second sensor associated with the vehicle relative to the point cloud data, wherein the second sensor has been pre-calibrated to the global coordinate system.

The computer program product described above is also configured to perform any of the operations/functions and may include any of the additional features/aspects of the exemplary embodiments of the invention described above with respect to the exemplary computer-implemented methods of the invention.

These and other features of the systems, methods, and non-transitory computer-readable media disclosed herein, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention.

Drawings

Certain features of various embodiments of the technology are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present technology will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings of which:

FIG. 1A illustrates a statically mapped object, such as a road sign in the environment through which a vehicle travels, which may be used, at least in part, to calibrate a vehicle sensor, such as a LiDAR, according to an exemplary embodiment of the present invention.

FIG. 1B illustrates an alternative environment in which a vehicle is traveling in which a static mapping object may be used in conjunction with calibration of vehicle sensors (such as LiDAR), according to an exemplary embodiment of the present invention.

FIG. 2 is a hybrid data flow and block diagram illustrating calibration of a first vehicle sensor, such as LiDAR, via a pre-calibrated second sensor, such as a GPS IMU, using a static mapping object present in the environment through which the vehicle is traveling, according to an exemplary embodiment of the present invention.

FIG. 3 is a process flow diagram of an illustrative method of calibrating a first vehicle sensor via a pre-calibrated second sensor using a static mapping object present in an environment through which the vehicle is traveling, in accordance with an exemplary embodiment of the present invention.

FIG. 4 is a process flow diagram of an illustrative method for identifying a set of environmental conditions associated with an environment through which a vehicle is traveling and determining whether the environmental conditions are optimal for initiating calibration of vehicle sensors in accordance with an exemplary embodiment of the present invention.

FIG. 5 is a process flow diagram of an illustrative method for performing calibration verification for vehicle sensors in accordance with an exemplary embodiment of the present invention.

Fig. 6 is a schematic block diagram illustrating an example network architecture configured to implement an example embodiment of the present invention.

Detailed Description

SUMMARY

In the following description, certain specific details are set forth in order to provide a thorough understanding of various embodiments of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these details. Moreover, although various embodiments of the invention are disclosed herein, many adaptations and modifications may be made within the scope of the invention in accordance with the common general knowledge of those skilled in the art. Such modifications include the substitution of known equivalents for any aspect of the invention in order to achieve the same result in substantially the same way.

Throughout this specification and claims, unless the context requires otherwise, the word "comprise", and variations such as "comprises" and "comprising", will be interpreted in an open, inclusive sense, that is, to mean "including, but not limited to. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, including the value defining the range, and each separate value is incorporated into the specification as if it were individually recited herein. Additionally, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. The phrases "at least one of", "at least one selected from the group consisting of" or "at least one selected from the group consisting of" and the like are to be interpreted in disjunctive terms (e.g., not interpreted as at least one of a and at least one of B).

Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, operation, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but may in some cases. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Typically, a vehicle may be equipped with a large number of on-board sensors. Such sensors may be disposed on the exterior or in the interior of the vehicle, and may include, but are not limited to, LiDAR sensors, radar, cameras, GPS receivers, sonar-based sensors, ultrasonic sensors, IMUs, accelerometers, gyroscopes, magnetometers, FIR sensors, and the like. Such sensors play a central role in the operation and operation of, for example, autonomous vehicles, semi-autonomous vehicles, and unmanned vehicles. As used herein, an autonomous vehicle may refer to a vehicle capable of operating in a fully autonomous driving mode in which manual input is not required for the vehicle to perform driving functions. In some exemplary embodiments, the autonomous vehicle may operate as an unmanned vehicle. For example, in some example embodiments, an autonomous vehicle may be operated without any occupants in the vehicle. In other exemplary embodiments, the autonomous vehicle may include one or more passive vehicle occupants (e.g., vehicle occupants without the ability to take over manual driving control of the vehicle). For example, such a vehicle occupant may be a passenger being transported by an autonomous vehicle operating in association with an autonomous vehicle ride sharing service. In other exemplary embodiments, the autonomous vehicle may include a vehicle occupant having the ability to provide input/commands to manually implement one or more driving functions of the autonomous vehicle. In other exemplary embodiments, the vehicle may be a semi-autonomous vehicle adapted to perform some, but not all, of the driving functions entirely autonomously. That is, a semi-autonomous vehicle may be able to operate fully autonomously under certain driving conditions (e.g., above a threshold speed, below a threshold amount of traffic density, etc.), but may require human operators to operate in other environments/scenarios in which there is no autonomous ability (e.g., driving on poorly marked roads). Although exemplary embodiments of the present invention may be described herein in connection with autonomous vehicles, it should be understood that embodiments of the present invention are applicable to vehicles having any level of automatic operation capability.

In an exemplary embodiment, the onboard sensors associated with the autonomous vehicle may include LiDAR, which may be used to detect objects (e.g., other vehicles, road signs, pedestrians, buildings, etc.) in the environment through which the vehicle is traveling. LiDAR may also be used to determine the relative distances between objects and vehicles in an environment. As another non-limiting example, radar may be used in conjunction with collision avoidance, adaptive cruise control, blind spot detection, assisted parking, and other vehicle applications. As yet another non-limiting example, the camera may be used to recognize, interpret, and/or identify objects captured in images or visual cues of the objects. The data collected from these sensors may be processed and used as input to an algorithm configured to make various autonomous driving decisions, including decisions regarding when to accelerate, when to decelerate, when to change direction, how much to accelerate, how much to decelerate, how much to change direction, etc.

In various exemplary embodiments of the invention, a large number of on-board sensors provide a continuous stream of sensor data that is in turn provided as input to algorithms that perform complex calculations to facilitate multiple operations required for safe autonomous vehicle operation, such as object detection, object classification/semantic segmentation, instance segmentation, object tracking, collision avoidance, vehicle navigation, vehicle acceleration and deceleration, and the like. To ensure that such operations are performed in accordance with the strict safety requirements of the autonomous vehicle, the calculations performed by the autonomous vehicle must have a high level of precision and accuracy, which in turn requires that the sensor data on which the calculations are based also exhibit a high level of precision and accuracy. The accuracy and precision of the sensor data, in turn, is a function of whether the sensor that captured such data was properly calibrated.

Discussed herein are technical solutions to solve technical problems related to conventional sensor calibration, in particular related to calibration of sensors associated with a vehicle such as an autonomous vehicle. Traditionally, calibration of vehicle sensors, such as LiDAR, is performed by operating the vehicle in a confined environment such that the LiDAR can capture point cloud data from specific objects present in the environment that is pre-selected to be used as a basis for LiDAR calibration to a global coordinate system. For example, in a conventional calibration scenario, an autonomous vehicle may be operated according to a predetermined travel path in a confined environment. During operation of the vehicle according to a predetermined travel path, the LiDAR sensor may periodically scan the restricted environment and emit light pulses that reflect off of preselected objects and are captured as point cloud data. For example, the preselected object may be a wall or other substantially flat structure deemed suitable for sensor calibration. Over time, the LiDAR may be calibrated based on the captured point cloud data until a desired calibration accuracy is achieved, that is, until the point cloud data representing the preselected object maintains a desired level of alignment with the fixed position of the preselected object within the global coordinate system.

However, such conventional methods for calibrating vehicle sensors, such as LiDAR, suffer from various technical drawbacks. In particular, this conventional approach requires the expenditure of additional resources in the form of additional time and energy consumption required to operate the vehicle for calibration purposes only. This additional resource consumption introduces opportunity costs because the consumed resources could have been used to dispatch vehicles to, for example, collect sensor data that can be used to refine static maps of the environment, improve autonomous driving capabilities, and the like.

Various embodiments of the present invention overcome the technical problems that arise specifically in the field of computer-based technology and more specifically in the field of autonomous vehicle technology. In particular, exemplary embodiments of the present invention provide a technical solution to the above-mentioned technical problems associated with conventional calibration techniques by providing systems, methods, non-transitory computer-readable media, techniques and methodologies for improving sensor calibration that do not require additional resource consumption.

More specifically, exemplary embodiments of the present invention provide calibration techniques whereby a first sensor (e.g., LiDAR) is calibrated to a global coordinate system via a second sensor (e.g., GPS IMU) that has been pre-calibrated to the global coordinate system. As shown in fig. 1A, in an exemplary embodiment, a vehicle 102, such as an autonomous vehicle, may be dispatched as it would normally be in an environment. More specifically, the vehicle 102 may traverse one or more road segments (e.g., road segment 104) in the environment. For ease of depiction, the road segment 104 and the direction of travel 110 of the vehicle 102 along the road segment 104 are depicted in fig. 1A as being substantially linear. However, it should be appreciated that the road segment 104 may form a portion of a larger road that includes any number of linear portions and/or non-linear portions having any curvature. For example, as shown in FIG. 1B, a road that includes the linear segments 104 shown in FIG. 1A may also include non-linear segments 118 having any type of curvature. Thus, the vehicle 102 may traverse various road segments in an environment having any linearity and/or non-linearity. Further, it should be understood that the vehicle travel direction 110 depicted in fig. 1A may vary between a range of allowable travel directions on the road segment 104, and that the travel path of the vehicle 102 along a road including, for example, the road segment 104 (fig. 1A) and the road segment 118 (fig. 1B) may include any number of turns, reversals in travel direction, deviations from a linear travel direction, and so forth.

The environment through which the vehicle 102 travels may include various static objects, such as signs present on one side of a road (e.g., speed limit signs 112, food/accommodation signs 114, etc.) or signs extending along or above a road (e.g., elevated highway signs 116). The signs encountered in the environment in which the vehicle 102 travels may vary in size, height, and shape. As will be described in more detail later in this disclosure, performing sensor calibration using static objects (such as statically mapped markers) over a range of sizes and heights may result in faster statistical convergence to a desired calibration accuracy. Exemplary markers 120, 122 having different sizes and heights are schematically illustrated in FIG. 1B. Although not depicted in fig. 1A and 1B, other examples of static objects in an environment may include, for example, physical structures such as buildings, monuments, landmarks, etc.; naturally occurring static objects such as trees, bushes, and the like; and others. As used herein, a static object refers to an object that is not designed for motion, either as a result of its own intent, such as a pedestrian, animal, or the like, or as a result of locomotive power provided to an object, such as a vehicle. However, static objects are not prohibited from undergoing some degree of motion due to externally applied forces (e.g., a sign or tree that moves slightly due to wind forces).

In the exemplary embodiment, vehicle 102 may be any object capable of generating locomotive power from an energy source. The energy source may be a liquid fuel, such as gasoline or Compressed Natural Gas (CNG); a hydrogen fuel cell; such as electricity from one or more batteries; solar energy; biomass; or any other suitable energy source. Vehicle 102 may be any type of vehicle including, but not limited to, an automobile, a truck, a bus, a motorcycle, an all-terrain vehicle, or others. The vehicle 102 may be a fully autonomous vehicle, an unmanned vehicle, a semi-autonomous vehicle, or the like.

In an exemplary embodiment, the vehicle 102 may include various onboard vehicle sensors. These sensors may include, for example, a LiDAR sensor 106, as well as various other sensors 108. Other sensors 108 may include, for example, inertial sensors (e.g., IMUs, accelerometers, gyroscopes, etc.); a thermal sensor; a humidity sensor; a vibration sensor; an image sensor; a radar; an ultrasonic sensor; and so on. For example, included among the other sensors 108 may be pre-calibrated sensors, such as GPS IMUs, via which LiDAR 106 may be calibrated to a global coordinate system in accordance with an exemplary embodiment of the present invention.

Various sensors on the vehicle 102 may be disposed outside and/or inside the vehicle 102. For example, LiDAR 106 may be disposed on the roof of vehicle 102. LiDAR 106 may be configured to periodically perform 360 degree scans of the surrounding environment, and as such, placing LiDAR 106 on the roof of vehicle 102 may optimize the amount of point cloud data that LiDAR 106 may capture. Various onboard vehicle sensors may be physically integrated with the vehicle 102 during manufacture of the vehicle 102, or may be attached or otherwise physically connected to the vehicle 102 after manufacture of the vehicle 102. The sensors may be communicatively coupled to one or more components (not shown) of the vehicle 102, such as an electronic control unit (ECC) of the vehicle 102; an on-board computer; or otherwise.

In some exemplary embodiments, LiDAR 106 may be provided as part of a sensor assembly that also includes one or more cameras. In some exemplary embodiments, the sensor assembly may also include other types of sensors, such as a GPS receiver or an IMU, among others. That is, in some exemplary embodiments, at least some of the LiDAR 106 and other sensors 108 may be provided as part of a combined sensor assembly. For example, the sensor assembly may be positioned on the roof of the vehicle 102. For example, in an example configuration, the LiDAR sensor 106 may be located in the center of the roof of the vehicle 102 and surrounded by a plurality of cameras positioned circumferentially around the LiDAR sensor 106.

In an exemplary embodiment, the LiDAR sensor 106 may periodically rotate through a scan path during which the LiDAR 106 may illuminate objects in the scanning environment with periodic pulses of light and measure differences in time-of-flight and wavelength of the reflected light to detect the presence of target objects and generate a 3D point cloud representation of the target illuminated by the pulses of light; determining a distance between the vehicle 102 and the target object; determine distances between various target objects, and the like. LiDAR 106 may present a horizontal scan path and/or a vertical scan path. More specifically, the LiDAR sensor 106 may generate 3D point cloud data (a set of data points in 3D space) that represents a target object that it has illuminated with light during a scan path of the light. The objects illuminated with light from LiDAR 106 may include any type of static object previously described; and one or more moving objects such as other vehicles, pedestrians, etc. In some exemplary embodiments, cameras positioned circumferentially around LiDAR 106 may capture image data that may be analyzed using trained classifiers or other machine learning techniques to perform object detection/perception to identify and classify objects present in the image data. It should be appreciated that the above-described configuration is merely illustrative, and that any number of LiDAR and/or other sensor/sensor assemblies may be disposed onboard the vehicle 102.

In an exemplary embodiment, as the vehicle 102 traverses the environment, a vehicle sensor (e.g., LiDAR 106) that requires calibration may capture point cloud data corresponding to static objects (such as signs) detected by the LiDAR 106 in the environment. The captured point cloud data may correspond to various static objects of different sizes, shapes, heights, and the like. The location of objects detected by LiDAR 106 may be determined. The location may be determined by, for example, a Global Positioning System (GPS) receiver associated with the vehicle 102. The location of the detected object may be determined relative to a global coordinate system. In some alternative exemplary embodiments, the location of the detected object within the global coordinate system may be determined relative to another vehicle sensor that has been calibrated to the global coordinate system.

In an exemplary embodiment, map data may be accessed. The map data may indicate the location of various static objects within the environment in the global coordinate system and the object type of each mapped static object. In an exemplary embodiment, the location of the detected static object may be determined to correspond to the location of the static mapping object in the map data, in which case the detected static object may be determined to be the same object of the same object type as the static mapping object, the location of the static mapping object in the map data corresponding to the determined location of the detected static object. Then, in an exemplary embodiment, it may be determined whether the determined type of static mapping object satisfies one or more criteria for calibrating the first sensor (e.g., LiDAR 106). For example, such criteria may include evaluating at least one of a shape feature, a size feature, or a surface feature of the statically mapped object relative to one or more features preselected to be optimal for calibrating the first sensor.

In particular, a similarity measure indicative of a degree of similarity between the size/shape/surface features of the statically mapped object and the preselected best feature may be determined and compared to a threshold. If the similarity metric satisfies a threshold (e.g., the similarity metric indicates at least a threshold level of similarity between the evaluated features of the static mapping object and the preselected best features for sensor calibration), the corresponding static mapping object may be deemed suitable for use in calibrating the first sensor. Depending on the implementation, a similarity metric that satisfies a threshold may include being strictly greater than the threshold; greater than or equal to a threshold; strictly less than a threshold; or a similarity measure less than or equal to a threshold. In general, a static mapping object having substantially regular boundaries and substantially flat surface features (e.g., regular polygon shaped markers) may be considered most suitable for sensor calibration.

After identifying a static mapping object suitable for sensor calibration, a calibration process may be initiated with respect to the captured point cloud data to calibrate the LiDAR 106 via a second pre-calibrated sensor (such as a GPS IMU or other inertial vehicle sensor) of the other sensors 108. For example, the calibration process may include determining a first transformation matrix from a first local coordinate system of LiDAR sensor 106 to a second local coordinate system of a pre-calibrated GPS IMU. Determining the first transformation matrix may include determining at least one rotation component and at least one translation component of the first transformation matrix. Because the second sensor 108 (e.g., GPS IMU) has been calibrated to the global coordinate system, the second transformation matrix from the local coordinate system of the second sensor to the global coordinate system is known. Thus, application of the first transformation matrix results in converting the location of a detected statically mapped object (e.g., represented by corresponding captured point cloud data) in the local coordinate system of a first sensor (e.g., LiDAR 106) to a corresponding location in the local coordinate system of a second sensor (e.g., GPS IMU). Application of the second transformation matrix then results in a conversion of the corresponding location in the local coordinate system of the second sensor to the location of the detected statically mapped object in the global coordinate system.

In an exemplary embodiment, the process for calibrating a first sensor (e.g., LiDAR 106) may make an iterative re-determination of the first transformation matrix until a desired calibration accuracy is achieved. More specifically, as the vehicle travels through the environment, the calibrated first sensor may capture sets of point cloud data representative of various statically mapped objects in the environment from various relative distances and perspectives between the first sensor and the static objects. The first transformation matrix may be iteratively re-determined for each such point cloud data set, and with each iterative re-determination, improved alignment between successively captured point cloud data corresponding to the detected statically mapped object is achieved, indicating improved accuracy of the calibration of the first sensor to the global coordinate system.

In an exemplary embodiment, the first transformation matrix is iteratively re-determined with respect to a plurality of detected static mapping objects in an environment in which the vehicle 102 is traveling until a calibration accuracy achieved by the first sensor (e.g., LiDAR 106) is within a desired calibration accuracy threshold. More specifically, in some exemplary embodiments, the calibration process is performed until the calibration error is less than an acceptable calibration error threshold, e.g., as represented by a deviation between a currently determined position of a detected static mapping object in the global coordinate system and a previously determined position of the detected static mapping object in the global coordinate system. The current determined location of the detected statically mapped object in the global coordinate system may be obtained by first applying a first transformation matrix to the object location in the local coordinate system of the first sensor to be calibrated (e.g., LiDAR 106) to obtain a corresponding location of the statically mapped object in the local coordinate system of the pre-calibrated second sensor 108 (e.g., GPS IMU), and then applying a known second transformation matrix to the location of the statically mapped object in the local coordinate system of the pre-calibrated second sensor to obtain the location of the statically mapped object in the global coordinate system. Similarly, a previously determined position of the statically mapped object in the global coordinate system may be obtained by first applying a previous iteration of the first transformation matrix and then applying a known second transformation matrix.

In some exemplary embodiments, calibration of a first sensor (e.g., LiDAR 106) may be initiated when it is determined that environmental conditions are determined to be optimal for calibration. For example, third party data indicative of environmental conditions related to the environment through which the vehicle 102 travels may be collected. Such data may include, for example, weather data, time of day data, data indicating the density of static mapping objects in the environment, and the like. Additionally or alternatively, sensor data indicative of environmental conditions may be captured. Such sensor data may be obtained from one or more of the other sensors 108 and may include, for example, data captured by a humidity sensor, a thermal sensor, or a vibration sensor, among others. The sensor data indicative of the environmental condition may also include data indicative of a quality of a GPS signal received by the GPS device, the IMU of which may be used as a basis for calibrating the first sensor; a vibration quantity detected by the GPS IMU; or otherwise. The data indicative of the environmental conditions may be evaluated against various criteria to determine whether an optimal set of conditions exists for initiating calibration of the sensor.

In some exemplary embodiments, after calibrating a first sensor (e.g., LiDAR 106) via a second sensor 108 (e.g., GPS IMU), a calibration verification process may be performed on the first sensor to ensure that the calibration accuracy of the first sensor is still within a desired calibration accuracy threshold. In an exemplary embodiment, the calibration verification process may be triggered in response to a vehicle event involving the vehicle 102, and/or may be performed periodically after an initial calibration of the first sensor. For example, vehicle events that may trigger calibration verification include vibration events that cause the vehicle to vibrate a threshold amount (e.g., travel over a speed bump or pothole); at least a threshold amount of force exerted on the first sensor determined from data captured by the force/pressure sensor (e.g., operating the vehicle 102 in high wind conditions); at least a threshold amount of heat present in or near the first sensor determined from data captured by the thermal sensors (e.g., operating the vehicle 102 in a high thermal condition); or otherwise. In some exemplary embodiments, the calibration verification of the first sensor may occur periodically: after a threshold amount of time has elapsed since the initial calibration or since the last calibration verification; after the vehicle 102 has traveled the threshold distance; after the vehicle 102 has been operating for a threshold amount of time; and so on.

In an exemplary embodiment, the calibration verification process may include determining a calibration error associated with a first sensor (e.g., LiDAR 106). As previously described, determining the calibration error may include, for example, determining a deviation between data representative of alignments of the statically mapped object detected in the environment of the vehicle 102 with different sets of point cloud data captured at different times (e.g., different relative distances/perspectives between the vehicle 102 and the detected static object). In other words, a calibration error of a first sensor (e.g., LiDAR 106) may be determined based on a deviation between a determined position of a detected statically mapped object within a global coordinate system (determined by applying a current iteration of a first transformation matrix to perform a transformation from the local coordinate system of the first sensor to the local coordinate system of a second pre-calibrated sensor (e.g., a GPS IMU), followed by applying a second known transformation matrix to perform a transformation from the local coordinate system of the second sensor to the global coordinate system) and a previously determined position of the detected statically mapped object within the global coordinate system (determined by applying a previous iteration of the first transformation matrix, followed by applying the second known transformation matrix).

It may then be determined whether the determined calibration error exceeds a threshold acceptable calibration error. The determination may involve: a determination is made whether the first sensor (e.g., LiDAR 106) maintains a calibration accuracy that is within a desired calibration accuracy threshold despite a calibration error. If the calibration error exceeds a threshold acceptable calibration error, recalibration of the first sensor may be initiated using the techniques described herein. That is, the first sensor may be recalibrated via a second pre-calibrated sensor, based on which calibration of the first sensor is initially performed.

Exemplary embodiments of the present invention provide a technical solution to the technical problems associated with conventional vehicle sensor calibration techniques. In particular, exemplary embodiments of the present invention provide improved vehicle sensor calibration techniques that enable calibration of a first vehicle sensor (such as LiDAR) via a second pre-calibrated vehicle sensor (such as a GPS IMU) without having to expend additional resources to perform the calibration separately. More specifically, according to an exemplary embodiment, a vehicle, such as an autonomous vehicle, may be dispatched into a driving environment, as is commonly done. Sensor data (e.g., point cloud data) representative of static objects detected in a vehicle environment may be captured. The map data may be accessed to identify those detected static objects that have been mapped in the map data. The statically mapped object may then be used to facilitate sensor calibration. Technical features of exemplary embodiments of the present invention include: capturing, by a first sensor to be calibrated, point cloud data representing static objects detected in an environment; identifying those detected objects as static mapping objects that meet criteria used in sensor calibration; and iteratively re-determining the first transformation matrix for transforming the detected position of the statically mapped object in the local coordinate system of the first sensor (as represented by the respective captured point cloud data) to a respective position in the local coordinate system of the second sensor that has been pre-calibrated to the global coordinate system.

According to an example embodiment, the first transformation matrix may be iteratively re-determined until a desired calibration accuracy is reached (e.g., the calibration error of the first sensor is less than a threshold acceptable calibration error). Since the second sensor has been pre-calibrated, a second transformation matrix for transforming the positions of the detected statically mapped objects in the local coordinate system of the second sensor to positions in the global coordinate system is known. In this manner, the iterative re-determination of the first transformation matrix until the desired calibration accuracy is reached allows the calibration of the position of the detected static mapping object in the local coordinate system of the first sensor to the global position of the detected static mapping object in the global coordinate system via the known second transformation matrix.

In this manner, a technical solution to the technical problems associated with conventional calibration techniques is achieved because a first sensor (e.g., LiDAR) may be calibrated via a pre-calibrated second sensor (e.g., GPS IMU) during operation of a vehicle that has been scheduled in a typical operating environment without having to expend additional resources (e.g., time, energy used to power the vehicle, etc.) to separately calibrate the first sensor, as required by conventional calibration techniques. The technical solution represents a technical improvement over conventional autonomous vehicle technology, in particular, an improvement over conventional technology for calibrating vehicle sensors of a vehicle, such as an autonomous vehicle.

Illustrative embodiments

FIG. 2 is a hybrid data flow and block diagram illustrating calibration of a first vehicle sensor, such as LiDAR, via a pre-calibrated second sensor, such as a GPS IMU, using a static mapping object detected in an environment through which the vehicle is traveling, according to an exemplary embodiment of the present invention. FIG. 3 is a process flow diagram of an illustrative method 300 for calibrating a first vehicle sensor via a pre-calibrated second sensor using a static mapping object detected in an environment in which the vehicle is traveling, in accordance with an exemplary embodiment of the present invention. FIG. 4 is a process flow diagram of an illustrative method 400 for identifying a set of environmental conditions associated with an environment through which a vehicle is traveling and determining whether the environmental conditions are optimal for initiating calibration of vehicle sensors in accordance with an exemplary embodiment of the present invention. Each of fig. 3 and 4 will be described below in conjunction with fig. 2.

Each operation of each of the methods 300, 400, and/or 500 described herein may be performed by one or more of the engines/program modules depicted in fig. 2 or fig. 6, the operations of which will be described in greater detail below. These engine/program modules may be implemented in any combination of hardware, software, and/or firmware. In certain example embodiments, one or more of these engine/program modules may be implemented, at least in part, as software and/or firmware modules comprising computer-executable instructions that, when executed by processing circuitry, cause one or more operations to be performed. In an exemplary embodiment, these engine/program modules may be custom computer-executable logic implemented within a custom computing machine, such as a custom FPGA or ASIC. Systems or devices described herein as being configured to implement exemplary embodiments of the invention may include one or more processing circuits, each of which may include one or more processing units or cores. The computer-executable instructions may include computer-executable program code that, when executed by the processing core, may cause input data contained in or referenced by the computer-executable program code to be accessed and processed by the processing core to produce output data.

Referring first to FIG. 2, a vehicle 202 is depicted. For example, the vehicle 202 may be a particular implementation of the vehicle 102. The vehicle 202 may be dispatched as it would normally be in an environment. More specifically, the vehicle 202 may travel one or more road segments that include any number of linear portions and/or non-linear portions having any curvature, as described with reference to the vehicle 102. Further, although the direction of travel 212 of the vehicle 202 is depicted in FIG. 2 as being substantially linear, it should be understood that the vehicle 202 may travel according to any one of a range of allowable directions of travel along a road, including making any number of turns, reversing the direction of travel, deviating from a linear direction of travel, and the like.

The environment through which the vehicle 202 passes may include various static objects, such as various types of signs 210 (e.g., signs appearing on the side of a roadway; signs extending along or over a roadway; etc.). The signs encountered in the environment in which the vehicle 202 travels may vary in size, height, and shape. According to an exemplary embodiment, using static mapping objects (such as detected markers) detected within the size and height range of the sensor calibration may result in faster statistical convergence to a desired calibration accuracy. Various other types of static objects 208 may also be present in the environment, and if statically mapped, may be used to perform sensor calibration according to the example embodiments described herein, including, for example, physical structures such as buildings, monuments, landmarks, and the like. In some exemplary embodiments, it may be most desirable to use static mapping landmarks because of their height differences in size and height and their substantially flat configuration. These features of a landmark may allow LiDAR to capture substantially uniform and dense point cloud data representing the landmark.

In an exemplary embodiment, the vehicle 202 may include various onboard vehicle sensors. These sensors may include, for example, a LiDAR sensor 204, as well as various other sensors 206. Other sensors 206 may include any of the types of sensors described above with reference to sensors 108. For example, included among the other sensors 206 may be pre-calibrated sensors, such as GPS IMUs, via which the LiDAR204 may be calibrated in accordance with an exemplary embodiment of the present invention.

Various sensors on the vehicle 202 may be disposed outside and/or inside the vehicle 202. For example, the LiDAR204 may be disposed on the roof of the vehicle 202. As described by LiDAR 106, LiDAR204 may be configured to periodically perform a 360 degree scan of the surrounding environment. Various onboard vehicle sensors may be physically integrated with the vehicle 202 during manufacture of the vehicle 202, or may be attached or otherwise physically connected to the vehicle 202 after manufacture of the vehicle 202. The sensors may be communicatively coupled to one or more components (not shown) of the vehicle 202, such as the ECC of the vehicle 202; an on-board computer; or otherwise.

In some exemplary embodiments, the LiDAR204 may be provided as part of a sensor assembly that also includes one or more cameras. In some exemplary embodiments, the sensor assembly may also include other types of sensors, such as a GPS receiver or an IMU, among others. That is, in some exemplary embodiments, at least some of the LiDAR204 and other sensors 206 may be provided as part of a combined sensor assembly. For example, the sensor assembly may be positioned on the roof of the vehicle 202. For example, in an example configuration, the LiDAR sensor 204 may be located in the center of the roof of the vehicle 202 and surrounded by a plurality of cameras positioned circumferentially around the LiDAR sensor 204. The LiDAR sensor 204 may be configured to generate 3D point cloud data (a set of data points in 3D space) that represents a target object that it has illuminated with light during a scan path of the light. In some exemplary embodiments, cameras positioned circumferentially around LiDAR204 may capture image data that may be analyzed using trained classifiers or other machine learning techniques to perform object detection/perception to identify and classify objects present in the image data. It should be appreciated that the above-described configuration is merely illustrative, and that any number of LiDAR and/or other sensor/sensor assemblies may be disposed onboard the vehicle 202.

Referring now to FIG. 3 in conjunction with FIG. 2, at block 302 of method 300, a first vehicle sensor (e.g., LiDAR 204) requiring calibration may capture point cloud data 214 corresponding to static objects detected in the environment. For example, the detected static object may be a logo 210 present in the environment or another type of static object, such as a physical structure 208 (e.g., a building, monument, etc.). In some exemplary embodiments, the point cloud data 214 captured at block 302 may include multiple sets of point cloud data representing the detected static object, where each set of point cloud data corresponds to a different relative distance and/or angle between the vehicle 202 and the detected static object. That is, as the vehicle 202 travels through the environment, the LiDAR204 may continuously capture point cloud data 214 representing light pulses emitted from the LiDAR204 and reflected back from static objects as the vehicle 202 comes closer to the detected static objects.

At block 304 of method 300, the position of the static object detected at block 302 may be determined relative to the global coordinate system. In an exemplary embodiment, the location of the detected static object may be determined by, for example, a GPS receiver associated with the vehicle 102. The GPS receiver may be included in the other sensors 206. In some alternative exemplary embodiments, the location of the detected object within the global coordinate system may be determined relative to another vehicle sensor that has been calibrated to the global coordinate system. The global coordinate system may be a coordinate system to which a first sensor (e.g., LiDAR 204) is to be calibrated according to example method 300, and to which a second sensor (e.g., GPS IMU) that is used as a basis for calibrating the first sensor has been calibrated. In an exemplary embodiment, the global coordinate system may provide an absolute reference system, wherein static objects have absolute positions, e.g., relative to the origin of the global coordinate system, which are not dependent on the reference system of the observer of the object.

In an exemplary embodiment, the indication 232 of the determined location of the detected static object may be provided as input to the static map correspondence engine 216. At block 306 of the method 300, the static map correspondence engine 216 may access the map data 234. In an exemplary embodiment, the map data 234 may be retrieved from one or more data stores 222. The map data 234 may indicate the mapped locations within the global coordinate system of various static objects within the environment (e.g., the environment over which the vehicle 202 is traveling), as well as the object types of the various static mapped objects. In an exemplary embodiment, the static map correspondence engine 216 may analyze the map data 234 to determine whether the calculated location of the detected static object matches the location of the static mapping object in the map data 234. If the static map correspondence engine 216 determines that the calculated location of the detected static object matches the location of the static mapping object in the map data 234, the detected static object may be determined to be the same object of the same object type as the static mapping object. In an exemplary embodiment, the static map correspondence engine 216 may send an indication to the calibration engine 218 that identifies the detected static object as a particular type of static mapping object (e.g., a flag).

Then, at block 308 of the method 300, the calibration engine 218 (or the static map correspondence engine 216) may determine whether the static mapping objects of the determined object type (i.e., the detected static objects determined at block 306 to have been statically mapped into the map data 234) satisfy one or more criteria for calibrating the first sensor (e.g., the LiDAR 204). For example, such criteria may include evaluating at least one of a shape feature, a size feature, or a surface feature of the statically mapped object relative to one or more features preselected to be optimal for calibrating the first sensor.

In particular, the calibration engine 218 (or the static map correspondence engine 216) may determine a similarity metric indicative of a degree of similarity between the size/shape/surface features of the static map object and the pre-selected best feature and compare the similarity metric to a threshold. If the similarity metric satisfies a threshold (e.g., the similarity metric indicates at least a threshold level of similarity between the evaluated features of the static mapping object and the preselected best features for sensor calibration), the corresponding static mapping object may be deemed suitable for use in calibrating the first sensor. In general, a static mapping object having substantially regular boundaries and substantially flat surface features (e.g., regular polygon shaped markers) may be considered most suitable for sensor calibration.

Upon determining that the static mapping object (e.g., object 210) detected in block 302 satisfies the criteria for sensor calibration, a calibration process may then be conducted with respect to the captured point cloud data 214 to calibrate the LiDAR204 via a second pre-calibrated sensor, such as a GPS IMU. As part of the calibration process, for example, at block 310 of method 300, the calibration engine 218 may determine a first transformation matrix from the first local coordinate system of the LiDAR sensor 204 to a second local coordinate system of the pre-calibrated GPS IMU. Determining the first transformation matrix may include determining at least one rotation component and at least one translation component of the first transformation matrix. Because the second sensor (e.g., GPS IMU) has been calibrated to the global coordinate system, a second transformation matrix from the local coordinate system to the global coordinate system of the second sensor is known. Thus, application of the first transformation matrix results in converting the location of a detected static object (e.g., represented by corresponding captured point cloud data 214) in the local coordinate system of a first sensor (e.g., LiDAR 204) to a corresponding location in the local coordinate system of a second sensor (e.g., GPS IMU). Application of the second transformation matrix then results in a conversion of the corresponding location in the local coordinate system of the second sensor into a location of the detected static object in the global coordinate system.

At block 312 of the method 300, the calibration engine 218 may determine whether the first transformation matrix results in a calibration of the first sensor (e.g., LiDAR 204) that is within a desired calibration accuracy threshold. More specifically, in an exemplary embodiment, the calibration engine 218 may determine whether the calibration error of the first sensor is less than an acceptable calibration error threshold, for example, as represented by a deviation between a currently determined position of the static object in the global coordinate system (detected at block 302) and a previously determined position of the static object detected in the global coordinate system. The current determined position of the detected static object in the global coordinate system may be obtained by first applying a first transformation matrix to the position of the static object in the local coordinate system of the first sensor to be calibrated (e.g., LiDAR 204) to obtain a corresponding position of the static object in the local coordinate system of the pre-calibrated second sensor (e.g., GPS IMU), and then applying a known second transformation matrix to the position of the static object in the local coordinate system of the pre-calibrated second sensor to obtain the position of the static object in the global coordinate system. Similarly, a previously determined position of the static object in the global coordinate system may be obtained by first applying a previous iteration of the first transformation matrix and then applying a known second transformation matrix. In some exemplary embodiments, the operations at block 310 may (at least initially) include determining at least two iterations of the first transformation matrix in order to perform the determination at block 312.

In response to a positive determination at block 312, the calibration process may end because the desired calibration accuracy of the LiDAR204 has been achieved. Thus, the current iteration of the first transformation matrix may be used as the final transformation matrix 230 for transforming object locations in the local coordinate system of the LiDAR204 to corresponding locations in the local coordinate system of the second sensor (e.g., GPS IMU), which in turn are transformed to locations in the global coordinate system by a second transformation matrix that is known based on a pre-calibration of the second sensor.

On the other hand, in response to a negative determination at block 312, the method 300 may proceed to block 314, where additional static objects may be detected in the environment in which the vehicle 202 is traveling, and additional point cloud data corresponding to the additional detected static objects may be captured by the LiDAR 204. Then, at block 316 of the method 300, the calibration engine 218 may re-determine the first transformation matrix with respect to the additional point cloud data. The method 300 may then return to block 312, where the calibration engine 218 may again determine whether the desired calibration accuracy has been achieved. The method 300 may proceed in this manner: the first transformation matrix is iteratively re-determined until a desired calibration accuracy of the first sensor (e.g., LiDAR 204) is reached. That is, in the exemplary embodiment, the first transformation matrix is iteratively re-determined with respect to a plurality of detected static objects in the environment over which vehicle 202 is traveling until the calibration accuracy achieved by the first sensor (e.g., LiDAR 204) is within a desired calibration accuracy threshold.

Although not explicitly depicted in fig. 3, it should be understood that in some exemplary embodiments, the operations at block 310 may include iteratively re-determining the first transformation matrix with respect to sets of point cloud data corresponding to the same detected static object, and the determination at block 312 may be made with respect to multiple iterations of the first transformation matrix, which is determined with respect to the same detected static mapping object. In other words, as the vehicle 202 travels through the environment, the calibrated first sensor (e.g., LiDAR 204) may capture sets of point cloud data representing particular detected statically mapped objects in the environment from various relative distances and perspectives between the first sensor and the objects. The first transformation matrix may be iteratively re-determined for each such point cloud data set, and successive iterative computations of the first transformation matrix relative to the same detected static object may be evaluated at block 312 to determine whether the desired calibration accuracy has been achieved. With re-determination of each iteration of the first transformation matrix, improved alignment between successively captured point cloud data corresponding to the detected statically mapped objects is achieved, indicating improved accuracy of the calibration of the first sensor to the global coordinate system. In some exemplary embodiments, the desired calibration accuracy may not be achieved based on iterative re-determination of the first transformation matrix relative to the point cloud data set corresponding to the same detected static mapping object, in which case the calibration process may continue through iterative re-determination of the first transformation matrix relative to the point cloud data set corresponding to one or more additional detected static objects. In an exemplary embodiment, the greater the variation in the shape, size, and/or height of the detected static mapping object based on which the sensor calibration is performed (as long as the similarity metric indicative of the level of similarity between the size/shape/surface features of the static mapping object and the optimal features of the sensor calibration meets the threshold), the faster the statistics of the calibration accuracy that reach within the desired calibration accuracy threshold converge.

Referring now to FIG. 4 in conjunction with FIG. 2, at block 402 of method 400, calibration initiation determination engine 220 may identify a need to calibrate a vehicle sensor (e.g., LiDAR 204). In some example embodiments, the LiDAR204 may require initial calibration. In other example embodiments, the LiDAR204 may need to be recalibrated as determined by a calibration verification process, for example, described later in this disclosure with reference to the example method 500 depicted in FIG. 5.

At block 404 of the method 400, the calibration initiation determination engine 220 may periodically access third party data 224 indicative of a first set of environmental conditions associated with the environment in which the vehicle 202 is traveling. For example, third party data 224 may be retrieved from data store 222. For example, the third party data 224 may include weather data; time of day data; data indicative of a density of static objects in the environment and/or other characteristics of the static objects in the environment; and others. For example, weather data may indicate various weather-related conditions present in the environment, such as temperature; climatic conditions (e.g., whether there is precipitation; if so, precipitation, visibility conditions, wind speed, cloud cover, etc.); and others. The time of day data may indicate a particular time of day, which may be associated with an expected visibility level of static objects in the environment. For example, static objects may be more easily seen when the expected amount of sunlight is greatest during the day. The data indicative of the density and/or other characteristics of the static objects in the environment may include data indicative of a total number of static objects present in the defined area of the environment; data indicating a plurality of static objects having an optimal size or shape in a defined area (e.g., due to their uniform shape, a flag may be more suitable for sensor calibration than a tree); data indicating a degree of change in height and/or shape of the static object; and others. It should be understood that the above examples of third party data 224 are merely illustrative and not exhaustive.

At block 406 of the method 400, the calibration initiation determination engine 220 may periodically monitor the sensor data 226 captured by one or more of the sensors 206. The sensor data 226 may be indicative of a second set of environmental conditions in the environment in which the vehicle 202 is traveling. For example, the sensor data 224 may include data captured by a humidity sensor, a thermal sensor, a vibration sensor, or the like. The humidity data may indicate a humidity level near a sensor (e.g., LiDAR 204) that requires calibration or recalibration; the heat data may indicate heat in or near the LiDAR 204; the vibration data may indicate a vibration level of one or more of the vehicle 202 (in general, and in particular, the LiDAR 204) or other sensors 206; and others. In some example embodiments, the sensor data 224 may also include data indicative of the quality of GPS signals received by a GPS device whose IMU may be used as a basis for calibrating the LiDAR 204; a vibration quantity detected by the GPS IMU; or otherwise.

At block 408 of the method 400, the calibration initiation determination engine 220 may determine whether the third party data 224 indicative of the first set of environmental conditions and/or the sensor data 226 indicative of the second set of environmental conditions satisfy one or more criteria for optimal calibration of a vehicle sensor (e.g., LiDAR 204). For example, a standard for optimal calibration of a vehicle may specify that a calibration procedure should be initiated when a particular set of environmental conditions exists, such as when the amount of sunlight in a particular time of day is expected to be maximum; during weather conditions that cause a level of visibility of static objects in the environment to at least reach a threshold level (e.g., below a threshold precipitation level, below a threshold fog level, etc.); during environmental conditions where the likelihood of impaired ability of the LiDAR204 to capture high-resolution point cloud data is mitigated (e.g., below a threshold wind speed around the LiDAR204, below a threshold temperature in or around the LiDAR204 housing, below a threshold humidity level in or around the LiDAR204 housing, below a threshold vibration level, etc.); and others. In some example embodiments, the criteria for optimal calibration of the LiDAR204 may additionally or alternatively require at least a threshold quality of GPS signals received by a GPS receiver whose IMU may be used as a basis for calibrating the LiDAR 204. It should be appreciated that the above examples of optimal criteria for initiating calibration or recalibration of LiDAR204 are merely illustrative and not exhaustive.

In response to a positive determination at block 406, the calibration initiation determination engine 220 may send a calibration initiation signal 228 to the calibration engine 218 to initiate calibration or recalibration of the LiDAR204 via a second sensor, such as a GPS IMU. On the other hand, in response to a negative determination at block 406, the method 400 may proceed iteratively from block 404, where additional environmental condition data (e.g., third party data 224, sensor data 226, etc.) may be periodically accessed at blocks 404 and 406 and evaluated against the criteria for optimal sensor calibration at block 408.

FIG. 5 is a process flow diagram of an illustrative method 500 for performing calibration verification for vehicle sensors in accordance with an exemplary embodiment of the present invention. In some example embodiments, after calibrating a first sensor (e.g., LiDAR 204) via a second sensor (e.g., GPS IMU), the method 500 may be performed with respect to the first sensor to ensure that the calibration accuracy of the first sensor remains within a desired calibration accuracy threshold.

At block 502 of method 500, a calibration verification process may be triggered in response to a vehicle event involving vehicle 202 and/or may be performed periodically after an initial calibration of the first sensor. For example, a vehicle event that may trigger calibration verification includes a vibration event that causes a threshold amount of vibration of the vehicle 202, the first sensor, and/or a second sensor via which the first sensor is calibrated (e.g., traveling over a speed bump or pothole); at least a threshold amount of force exerted on the first sensor determined from data captured by the force/pressure sensor (e.g., operating the vehicle 202 in high wind conditions); at least a threshold amount of heat present in or near the first sensor determined from data captured by the thermal sensors (e.g., operating the vehicle 202 in a high thermal condition); or otherwise. In some exemplary embodiments, the calibration verification of the first sensor may occur periodically: after a threshold amount of time has elapsed since the initial calibration or since the last calibration verification; after the vehicle 202 has traveled the threshold distance; after the vehicle 202 has been operating for a threshold amount of time; and so on.

In response to a triggering vehicle event or based on a predetermined periodicity, a calibration error associated with a first sensor (e.g., LiDAR 204) may be determined at block 502. As previously described, determining the calibration error may include, for example, determining a deviation between data representative of alignments of the detected static mapping object in the environment of the vehicle 202 with different sets of point cloud data captured at different times (e.g., different relative distances/perspectives between the vehicle 202 and the detected static mapping object). In other words, a calibration error of a first sensor (e.g., LiDAR 204) may be determined based on a deviation between a determined position of a detected statically mapped object within a global coordinate system (determined by applying a current iteration of a first transformation matrix to perform a transformation from the local coordinate system of the first sensor to the local coordinate system of a second pre-calibrated sensor (e.g., GPS IMU), followed by applying a second known transformation matrix to perform a transformation from the local coordinate system of the second sensor to the global coordinate system) and a previously determined position of the detected statically mapped object within the global coordinate system (determined by applying a previous iteration of the first transformation matrix, followed by applying the second known transformation matrix).

At block 504 of the method 500, it may be determined whether the determined calibration error exceeds a threshold acceptable calibration error. The determination may involve: a determination is made whether the first sensor (e.g., LiDAR 204) maintains a calibration accuracy that is within a desired calibration accuracy threshold despite a calibration error. If the calibration error exceeds the threshold acceptable calibration error (a positive determination at block 504), recalibration of the first sensor may be initiated at block 506 of method 500 using techniques described herein (e.g., the example method 300 of fig. 3). That is, the first sensor may be recalibrated via a second pre-calibrated sensor, based on which the initial calibration of the first sensor is performed. On the other hand, in response to a negative determination at block 504, method 500 may be iteratively conducted from block 502.

Hardware implementation

Fig. 6 is a schematic block diagram illustrating an example network architecture 600 configured to implement an example embodiment of the present invention. The networking architecture 600 may include one or more special-purpose computing devices 602 communicatively coupled to various sensors 604 via one or more networks 606. The sensors 604 may include any type of on-board vehicle sensor previously described, including but not limited to LiDAR sensors (e.g., LiDAR 106, LiDAR 204), radar, cameras, GPS receivers, sonar-based sensors, ultrasonic sensors, IMUs, accelerometers, gyroscopes, magnetometers, FIR sensors, and the like. In an exemplary embodiment, the sensors 604 may include onboard sensors disposed on or in the exterior of a vehicle (e.g., vehicle 102, vehicle 202), such as an autonomous vehicle. The dedicated computing device 602 may comprise a device integrated with the vehicle and may receive sensor data from the sensors 604 via a local network connection (e.g., WiFi, bluetooth, Dedicated Short Range Communication (DSRC), etc.). In other example embodiments, the dedicated computing device 602 may be located remotely from the vehicle and may receive sensor data from the sensors 604 via one or more remote networks.

The special purpose computing device 602 may be hardwired to perform the techniques of the exemplary embodiments of the present invention; may include circuitry or digital electronics permanently programmed to perform the techniques, such as one or more ASICs or FPGAs; and/or may include one or more hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage devices, or a combination thereof. The special purpose computing device 602 may also combine custom hard-wired logic, an ASIC, or an FPGA with custom programming to accomplish this technique. In other example embodiments, one or more sensors 604 (e.g., LiDAR) may include a corresponding custom processing unit (e.g., ASIC, FPGA, etc.) configured to perform techniques according to example embodiments of the present invention. The special purpose computing device 602 may be a desktop computer system, a server computer system, a portable computer system, a handheld device, a networked device, or any other device or combination of devices that incorporate hardwired and/or programmed logic to implement the techniques.

The dedicated computing device may generally be controlled and coordinated by operating system software 620, such as iOS, Android, Chrome OS, Windows XP, Windows Vista, Windows 7, Windows 8, Windows Server, Windows CE, Unix, Linux, SunOS, Solaris, iOS, Blackberry OS, VxWorks, or other compatible operating systems. In other embodiments, the computing device 602 may be controlled by a proprietary operating system. Operating system software 620 may control and schedule computer processes for execution; performing memory management; providing file system, networking and I/O services; and providing user interface functionality, such as a graphical user interface ("GUI").

Although computing device 602 and/or sensor 604 may be described herein in the singular, it should be understood that multiple instances of any such means may be provided and that the functionality described in connection with any particular means may be distributed across multiple instances of that means. In certain example embodiments, the functionality described herein in connection with any given component of the architecture 600 may be distributed among multiple components of the architecture 600. For example, at least a portion of the functionality described as being provided by computing device 602 may be distributed across a plurality of such computing devices 602.

The network 606 may include, but is not limited to, any one or more different types of communication networks, such as a wired network, a public network (e.g., the internet), a private network (e.g., a frame relay network), a wireless network, a cellular network, a telephone network (e.g., a public switched telephone network), or any other suitable private or public packet-switched or circuit-switched network. Network 606 may have any suitable communication range associated therewith and may include, for example, a global network (e.g., the internet), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a Local Area Network (LAN), or a Personal Area Network (PAN). Additionally, the network 606 may include communication links and associated networking equipment (e.g., link layer switches, routers, etc.) to transport network traffic over any suitable type of medium, including but not limited to coaxial cable, twisted pair (e.g., twisted copper pair), optical fiber, hybrid coaxial fiber (HFC) medium, microwave medium, radio frequency communication medium, satellite communication medium, or any combination thereof.

In the illustrative configuration, the computing device 602 may include one or more processors 608, one or more memory devices 610 (collectively referred to herein as memory 610), one or more input/output ("I/O") interfaces 612, one or more network interfaces 616, and a data storage device 618. Computing device 602 may also include one or more buses 616 that functionally couple various components of computing device 602. The data storage device may store one or more engines, program modules, components, and the like, including, but not limited to, a static map correspondence engine 624, a calibration engine 626, and a calibration initiation determination engine 628. Each of the engines/components depicted in fig. 6 may include logic for performing any of the previously described processes or tasks in conjunction with the correspondingly named engine/component. In certain example embodiments, any of the depicted engines/components may be implemented in hardwired circuitry within a digital electronic device, such as one or more ASICs or FPGAs, that are persistently programmed to perform the corresponding techniques.

Bus 616 may include at least one of a system bus, a memory bus, an address bus, or a message bus, and may permit information (e.g., data (including computer executable code), signaling, etc.) to be exchanged between various components of computing device 602. The bus 616 may include, but is not limited to, a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and the like. The bus 616 may be associated with any suitable bus architecture, including but not limited to Industry Standard Architecture (ISA), Micro Channel Architecture (MCA), enhanced ISA (eisa), Video Electronics Standards Association (VESA), Accelerated Graphics Port (AGP), Peripheral Component Interconnect (PCI), PCI-Express, international personal computer memory card association (PCMCIA), Universal Serial Bus (USB), and the like.

The memory 610 may include: volatile memory (memory that maintains its state when powered), such as Random Access Memory (RAM); and/or non-volatile memory (memory that maintains its state even when not powered), such as Read Only Memory (ROM), flash memory, ferroelectric ram (fram), and so forth. Persistent data storage, as that term is used herein, may include non-volatile memory. In some example embodiments, volatile memory may implement faster read/write access than non-volatile memory. However, in certain other example embodiments, certain types of non-volatile memory (e.g., FRAM) may implement faster read/write access than certain types of volatile memory.

In various embodiments, memory 610 may include a variety of different types of memory, such as various types of Static Random Access Memory (SRAM), various types of Dynamic Random Access Memory (DRAM), various types of non-alterable ROM, and/or writable variants of ROM, such as electrically erasable programmable read-only memory (EEPROM), flash memory, and so forth. Memory 610 may include main memory as well as various forms of cache memory such as an instruction cache, a data cache, a Translation Lookaside Buffer (TLB), and the like. Additionally, a cache memory, such as a data cache, may be a multi-level cache organized in a hierarchy of one or more cache levels (L1, L2, etc.). In an example embodiment, memory 610 may include data storage devices 106(1) -106(P) and/or data storage device 120 depicted in fig. 1. Alternatively, data storage devices 106(1) -106(P) may be hard disk storage devices that form part of data storage device 618 and/or data storage device 120 may be in the form of RAM or cache memory provided as part of FOV semantic computation machine 626 itself.

Data storage device 618 may include removable storage devices and/or non-removable storage devices, including but not limited to magnetic storage devices, optical storage devices, and/or tape storage devices. Data storage device 618 may provide non-volatile storage of computer-executable instructions and other data. Memory 610 and data storage device 618 (removable and/or non-removable) are examples of computer-readable storage media (CRSM) as the term is used herein. The data storage device 618 may store computer-executable code, instructions, or the like, that may be loaded into the memory 610 and executed by the processor 608 to cause the processor 608 to perform or initiate various operations. The data storage device 618 may additionally store data that may be copied to the memory 610 for use by the processor 608 in executing computer-executable instructions. Further, output data resulting from execution of the computer-executable instructions by processor 608 may be initially stored in memory 610 and may ultimately be copied to data storage device 618 for non-volatile storage.

More specifically, the data storage 618 may store one or more operating systems (O/ss) 620 and one or more database management systems (DBMS)622 configured to access the memory 610 and/or one or more external data stores (not depicted), potentially via one or more of the networks 606. Further, the data storage device 618 may also store one or more program modules, applications, engines, computer-executable code or scripts, or the like, including, for example, a static map correspondence engine 624, a calibration engine 626, and a calibration initiation determination engine 628. Any of the engines/components depicted in fig. 6 may be implemented as software and/or firmware, including computer-executable instructions (e.g., computer-executable program code) that are loadable into the memory 610 and executed by one or more of the processors 608 to perform any of the techniques described above in connection with the respectively named modules/engines.

Although not depicted in fig. 6, data storage 618 may also store various types of data utilized by the engines/components of computing device 602. Such data may include, but is not limited to, sensor data, transformation matrix data, static map data, or the like. Any data stored in the data storage device 618 may be loaded into the memory 610 for use by the processor 608 in executing computer-executable program code. Additionally, any data stored in data storage 618 may potentially be stored in one or more external data stores that are accessible via DBMS 622 and loadable into memory 610 for use by processor 608 in executing computer-executable instructions/program code.

The processor 608 may be configured to access the memory 610 and execute the computer-executable instructions/program code loaded therein. For example, the processor 608 may be configured to execute computer-executable instructions/program code of various engines/components of the computing device 602 to cause or facilitate performing various operations in accordance with one or more embodiments of the present invention. Processor 608 may include any suitable processing unit capable of accepting data as input, processing the input data according to stored computer-executable instructions, and generating output data. Processor 608 may include any type of suitable processing unit, including but not limited to a central processing unit, microprocessor, Reduced Instruction Set Computer (RISC) microprocessor, Complex Instruction Set Computer (CISC) microprocessor, microcontroller, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), system on chip (SoC), Digital Signal Processor (DSP), or the like. Additionally, the processor 608 may have any suitable micro-architectural design including any number of constituent components, such as, for example, registers, multiplexers, arithmetic logic units, cache controllers for controlling read/write operations to cache memory, branch predictors, and the like. The micro-architectural design of the processor 608 may be enabled to support any of a variety of instruction sets.

Referring now to other illustrative components depicted as being stored in the data storage device 618, the O/S620 may be loaded from the data storage device 618 into the memory 610 and may provide an interface between the computing apparatus 602 and other application software executing on the hardware resources of the computing apparatus 602. More specifically, O/S620 may include a set of computer-executable instructions for managing hardware resources of computing device 602 and for providing common services to other applications. In certain example embodiments, the O/S620 may include or otherwise control execution of one or more of the engine/program modules stored in the data storage device 618. O/S620 may comprise any operating system now known or later developed, including but not limited to any server operating system, any mainframe operating system, or any other proprietary or non-proprietary operating system.

DBMS 622 may be loaded into memory 610 and may support functionality for accessing, retrieving, storing, and/or manipulating data stored in memory 610, data stored in data storage device 618, and/or data stored in an external data store (not shown in fig. 6). The DBMS 622 may use any of a variety of database models (e.g., relational models, object models, etc.) and may support any of a variety of query languages. DBMS 622 may access data represented in one or more data patterns and stored in any suitable data store. The data stores accessible by the computing device 602 via the DBMS 622 may include, but are not limited to, databases (e.g., relational, object-oriented, etc.), file systems, flat files, distributed data stores in which data is stored on more than one node of a computer network, peer-to-peer network data stores, and the like.

Referring now to other illustrative components of computing device 602, input/output (I/O) interface 612 may facilitate computing device 602 receiving input information from one or more I/O devices and outputting information from computing device 602 to one or more I/O devices. The I/O devices may include any of a variety of components, such as a display or display screen having a touch surface or touch screen; an audio output device for producing sound, such as a speaker; an audio capture device, such as a microphone; an image and/or video capture device, such as a camera; a haptic unit, etc. Any of these components may be integrated into the computing device 602 or may be separate therefrom. The I/O devices may also include, for example, any number of peripheral devices, such as data storage devices, printing devices, and the like.

The I/O interface 612 may also include interfaces for external peripheral device connections, such as Universal Serial Bus (USB), FireWire, Thunderbolt, ethernet ports, or other connection protocols that may connect to one or more networks. The I/O interface 612 may also include a connection to one or more antennas to connect to one or more networks via a Wireless Local Area Network (WLAN) radio, such as a Wi-Fi radio, bluetooth, and/or a wireless network radio, such as a radio capable of communicating with a wireless communication network, such as a Long Term Evolution (LTE) network, WiMAX network, 3G network, and so forth.

Computing device 602 may also include one or more network interfaces 614 via which computing device 602 may communicate with any of a variety of other systems, platforms, networks, devices, and the like. Network interface 614 may enable communication with, for example, sensor 606 and/or one or more other devices via one or more of networks 606. In an example embodiment, network interface 614 provides a two-way data communication coupling to one or more network links connected to one or more of networks 606. For example, network interface 614 may include an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another non-limiting example, network interface 614 may include a Local Area Network (LAN) card to provide a data communication connection to a compatible LAN (or Wide Area Network (WAN) component in communication with a WAN). Wireless links may also be implemented. In any such implementation, network interface 614 may send and receive electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

Network links typically provide data communication through one or more networks to other data devices. For example, the network link may provide a connection through a local area network to a host computer or to data equipment operated by an Internet Service Provider (ISP). The ISP in turn may provide data communication services through the worldwide packet data communication network now commonly referred to as the "internet". Both local area networks and the internet use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks 606, which carry the digital data to and from the computing device 602, and the signals on the network links and through the network interface 614 are example forms of transmission media. In an example embodiment, computing device 602 may send messages and receive data, including program code, through the network 606, the network links, and the network interface 614. For example, in the Internet example, a server might transmit a requested code for an application program through the Internet, an ISP, local network and network interface 614. The received code may be executed by processor 608 as it is received, and/or stored in data storage device 618, or other non-volatile storage for later execution.

It should be appreciated that the engines depicted in FIG. 6 as part of computing device 602 are merely illustrative and not exhaustive. In particular, the functionality can be modular in any suitable manner such that processes described as being supported by any particular engine can alternatively be distributed across multiple engines, program modules, components, etc., or performed by different engines, program modules, components, etc. Additionally, in some embodiments one or more of the depicted engines may or may not be present, while in other embodiments additional engines not depicted may be present and may support the described functionality and/or at least a portion of the additional functionality. Additionally, various engines, program modules, scripts, plug-ins, Application Programming Interfaces (APIs), or any other suitable computer-executable code hosted locally on computing device 602 and/or hosted on other computing devices (e.g., 602) accessible via one or more of networks 606 may be provided to support the functionality provided by the engines depicted in fig. 6 and/or in addition to or in lieu of the functionality. Additionally, engines supporting the functionality described herein may be implemented at least in part in hardware and/or firmware and may execute on any number of computing devices 602 according to any suitable computing model (such as, for example, a client-server model, a peer-to-peer model, etc.).

It should also be understood that the computing device 602 may include alternative and/or additional hardware, software, and/or firmware components than those described or depicted without departing from the scope of the present invention. More specifically, it should be appreciated that the software, firmware, and/or hardware components depicted as forming part of computing device 602 are merely illustrative, and in various embodiments, some components may or may not be present, or additional components may be provided. It should also be appreciated that, in various embodiments, each of the depicted and described engines represents a logical partition of supported functionality. The logical partitions are depicted for ease of explanation of functionality and may or may not represent software, hardware, and/or firmware structures for implementing the functionality.

Generally, the terms engine, program module, and the like as used herein refer to logic embodied in hardware, firmware, and/or circuitry, or to a collection of software instructions written in a programming language such as Java, C, or C + +, possibly with entry and exit points. The software engine/module may be compiled and linked to an executable program, installed in a dynamically linked library, or may be written in an interpreted programming language such as, for example, BASIC, Perl, or Python. It will be appreciated that the software engines/modules may be invoked from other engines/modules or from themselves, and/or may be invoked in response to a detected event or interrupt. A software engine/module configured to execute on a computing device may be provided on a computer readable medium such as a compact disk, digital video disk, flash memory drive, magnetic disk, or any other tangible medium, or downloaded as digital (and may be initially stored in a compressed or installable format requiring installation, decompression, or decryption prior to execution). Such software code may be stored, partially or wholly, on a memory device of an executing computing device for execution by the computing device. "open source" software refers to source code that may be distributed as source code and/or in compiled form, obtained in well-known and indexed fashion, and optionally with permissions to allow modifications and derivative works. The software instructions may be embedded in firmware and stored, for example, on a flash memory such as an erasable programmable read-only memory (EPROM). It will also be appreciated that the hardware modules/engines may comprise connected logic units such as gates and flip-flops, and/or may also comprise programmable units such as programmable gate arrays or processors.

Exemplary embodiments are described herein as including engines or program modules. Such engine/program modules may constitute a software engine (e.g., code embodied on a machine-readable medium) or a hardware engine. A "hardware engine" is a tangible unit capable of performing certain operations and may be configured or arranged in a particular physical manner. In various example embodiments, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware engines (e.g., a processor or a set of processors) of a computer system may be configured by software (e.g., an application or application portion) as a hardware engine that operates to perform certain operations as described herein.

In some embodiments, the hardware engine may be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware engine may comprise dedicated circuitry or logic that is persistently configured to perform certain operations. For example, the hardware engine may be a special purpose processor, such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC). The hardware engine may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware engine may include a general-purpose processor or other programmable processor configured by software, in which case the configured processor becomes a specific machine uniquely customized to perform the configured functions and no longer constitutes a general-purpose processor. It will be appreciated that the decision to mechanically implement a hardware engine in a dedicated and persistently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations.

Thus, the terms "engine" or "program module" should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. Given the embodiments in which the hardware engines are temporarily configured (e.g., programmed), each of the hardware engines need not be configured or instantiated at any one time. For example, where the hardware engine comprises a general-purpose processor configured by software as a special-purpose processor, the general-purpose processor may be configured separately as different special-purpose processors (e.g., comprising different hardware engines) at different times. The software may configure one or more particular processors accordingly, e.g., to configure a particular hardware engine at a given time, and to configure different hardware engines at different times.

The hardware engine may provide information to and receive information from other hardware engines. Thus, the described hardware engines may be considered to be communicatively coupled. Where multiple hardware engines are present simultaneously, communication may be achieved through signaling (e.g., through appropriate circuitry and buses) between or among two or more of the hardware engines. In embodiments where multiple hardware engines are configured or instantiated at different times, communication between such hardware engines may be accomplished, for example, through storage and retrieval of information in memory structures accessible to the multiple hardware engines. For example, a hardware engine may perform an operation and store the output of the operation in a memory device communicatively coupled thereto. Another hardware engine may then access the memory device at a later time to retrieve and process the stored output. The hardware engine may also initiate communication with an input or output device and may operate on a resource (e.g., a set of information).

Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily configured or permanently configured, such processors may constitute an implementation of a hardware engine. Similarly, the methods described herein may be at least partially processor-implemented, where one or more particular processors are examples of hardware. Further, the one or more processors may also support the execution of related operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a set of computers (as an example of machines including processors), where the operations are accessible via a network (e.g., the internet) and via one or more appropriate interfaces (e.g., APIs).

Execution of certain operations of the example methods described herein may be distributed among multiple processors, not only residing within a single machine, but also being deployed across multiple machines. In some example embodiments, the processors may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processors may be distributed across multiple geographic locations.

The present invention may be embodied as systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to perform aspects of the invention.

A computer-readable storage medium, as that term is used herein, is in the form of a non-transitory medium and may be any tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. Computer-readable storage media, and more generally, non-transitory media, may include non-volatile media and/or volatile media. A non-exhaustive list of more specific examples of the computer readable storage medium includes: portable computer floppy disks, such as a floppy disk or a flexible disk; a hard disk; random Access Memory (RAM), Read Only Memory (ROM), erasable programmable read only memory (EPROM or flash memory), Static Random Access Memory (SRAM), or any other memory chip or cartridge; portable compact disc read only memory (CD-ROM); digital Versatile Disks (DVDs); a storage rod; a solid state drive; magnetic tape or any other magnetic data storage medium; mechanically encoded means such as a punch card or any physical medium with a pattern of holes or a raised structure in a groove having instructions recorded thereon; any networked version of the above; and any suitable combination of the foregoing.

The non-transitory medium is different from the transmission medium, and thus, as used herein, a computer-readable storage medium should not be understood to be a transitory signal in itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or an electrical signal transmitted through a wire. However, non-transitory media may operate in conjunction with transmission media. In particular, transmission media may participate in transferring information between non-transitory media. For example, transmission media may include coaxial cables, copper wire and/or fiber optics, including the wires that comprise at least some of buses 602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infrared data communications.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device via a network, such as the internet, a Local Area Network (LAN), a Wide Area Network (WAN), and/or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing/processing device.

Computer-readable program instructions for carrying out operations of the present invention may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a LAN or a WAN, or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider (ISP)). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, FPGAs, or Programmable Logic Arrays (PLAs), may execute computer-readable program instructions to perform aspects of the present invention by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable storage medium having the instructions stored therein comprises an article of manufacture including instructions which implement an aspect of the function/act specified in the flowchart and/or block diagram block or blocks. The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the invention. Additionally, in some implementations, certain method or process blocks may be omitted. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states associated therewith can be performed in other suitable sequences. For example, described blocks or states may be performed in an order different than that specifically disclosed, or multiple blocks or states may be combined in a single block or state. The example blocks or states may be performed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The example systems and components described herein may be configured differently than described. For example, elements may be added to, removed from, or rearranged compared to the disclosed example embodiments.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible embodiments of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative embodiments, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

It should be emphasized that many variations and modifications may be made to the above-described embodiments, and the elements of the embodiments are to be understood as being among other example embodiments of the invention. All such modifications and variations are intended to be included herein within the scope of this disclosure. Although example embodiments of the invention may be referred to herein, individually or collectively, by the term "invention" merely for convenience and without limiting the scope of the invention to any single disclosure or concept if more than one is in fact disclosed. The foregoing description details certain embodiments of the invention. It should be understood, however, that no matter how detailed the foregoing appears in text, the invention can be practiced in many ways. It should be noted that the use of particular terminology when describing certain features or aspects of the invention does not imply that the terminology is being redefined herein to be restricted to including any specific characteristics of the features or aspects of the invention with which that terminology is associated.

The embodiments illustrated herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. The particular embodiments are therefore not to be considered in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

Although the invention has been described in detail for the purpose of illustration based on what is currently considered to be the most practical and preferred implementations, it is to be understood that such detail is solely for that purpose and that the invention is not limited to the disclosed implementations, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present invention contemplates that, to the extent possible, one or more features of any embodiment can be combined with one or more features of any other embodiment.

As used herein, the term "or" may be interpreted in an inclusive or exclusive sense. Further, multiple instances of a resource, operation, or structure described herein as a single instance may be provided. In addition, the boundaries between the various resources, operations, program modules, engines, and/or data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of various embodiments of the invention. In general, structures and functionality presented as separate resources in the example configurations may be implemented as a combined structure or resource. Similarly, structures and functionality presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements may fall within the scope of the embodiments of the invention as expressed in the claims that follow. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Conditional language such as "can," "may," "might," or "may" is generally intended to convey that certain embodiments include, but other embodiments do not include, certain features, elements and/or steps, unless expressly stated otherwise, or otherwise understood within the context as used. Thus, such conditional language is not generally intended to imply that features, elements, and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether such features, elements, and/or steps are included or are to be performed in any particular embodiment. Further, it will be appreciated that any operation, element, component, data, etc., described herein as being based on another operation, element, component, data, etc., may additionally be based on one or more other operations, elements, components, data, etc. Thus, the phrase "based on" or variations thereof should be interpreted as "based, at least in part, on.

Claims (20)

1. A computer-implemented method for calibrating a first sensor associated with a vehicle using a static mapping object, the method comprising:

capturing, via the first sensor, point cloud data, at least a portion of which is associated with an object in an environment through which the vehicle is traveling;

determining that the object is a static mapping object;

determining that the static mapping object satisfies one or more criteria for calibrating the first sensor; and

calibrating the first sensor to a global coordinate system via a second sensor associated with the vehicle relative to the point cloud data, wherein the second sensor has been pre-calibrated to the global coordinate system.

2. The computer-implemented method of claim 1, wherein determining that the object is the static mapping object comprises:

determining a location of the object within the global coordinate system;

retrieving map data of the environment through which the vehicle is traveling; and

determining that the location of the object corresponds to a location of the static mapping object in the map data.

3. The computer-implemented method of claim 2, wherein calibrating the first sensor to the global coordinate system comprises:

determining a transformation matrix from a first local coordinate system of the first sensor to a second local coordinate system of the second sensor, wherein the transformation matrix comprises at least one rotational component and at least one translational component of the transformation matrix.

4. The computer-implemented method of claim 3, wherein the at least one rotational component and at least one translational component align at least a portion of the point cloud data with the position of the static mapping object within the global coordinate system as a relative position of the vehicle with respect to the static mapping object changes.

5. The computer-implemented method of claim 3, wherein calibrating the first sensor comprises determining whether the transformation matrix results in a calibration accuracy of the first sensor being within a calibration accuracy threshold.

6. The computer-implemented method of claim 5, wherein the transformation matrix is a first transformation matrix, and wherein calibrating the first sensor to the global coordinate system further comprises:

determining a location of the object in the first local coordinate system based at least in part on at least a portion of the point cloud data associated with the object;

applying the first transformation matrix to a position of the object in the first local coordinate system to obtain a corresponding position in the second local coordinate system;

applying a second transformation matrix to corresponding locations in the second local coordinate system to obtain calibrated locations of the object in the global coordinate system; and

determining the calibration accuracy of the calibration of the first sensor based at least in part on a deviation between the calibrated location of the object in the global coordinate system obtained via applying the first and second transformation matrices and a location of the static mapping object in the map data.

7. The computer-implemented method of claim 6, wherein determining whether the transformation matrix causes a calibration accuracy of the first sensor to be within a calibration accuracy threshold comprises determining that the calibration accuracy of the first sensor is not within the calibration accuracy threshold, and wherein at least a portion of the point cloud data associated with the object is a first portion of the point cloud data and the static mapping object is a first static mapping object, the method further comprising:

determining that a second object in the environment through which the vehicle is traveling is a second statically mapped object, a second portion of the point cloud data being associated with the second statically mapped object;

re-determining the transformation matrix relative to the second portion of the point cloud data; and

re-determining the calibration accuracy of the first sensor based at least in part on the re-determined transformation matrix.

8. The computer-implemented method of claim 1, wherein determining that the static mapping object satisfies the one or more criteria for calibrating the first sensor comprises:

determining at least one of a shape feature, a size feature, or a surface feature of the statically mapped object;

determining a similarity metric indicative of a level of similarity between at least one of a shape feature, a size feature, or a surface feature of the statically mapped object and one or more features preselected to be optimal for calibrating the first sensor; and

determining that the similarity metric satisfies a threshold.

9. The computer-implemented method of claim 1, further comprising:

after calibrating the first sensor, determining a calibration error associated with the first sensor;

determining that the calibration error exceeds a threshold acceptable calibration error; and

recalibrating the first sensor at least partially in response to determining that the calibration error exceeds the threshold acceptable calibration error.

10. The computer-implemented method of claim 1, further comprising, prior to calibrating the first sensor:

identifying data indicative of a set of environmental conditions in the environment through which the vehicle is traveling, wherein the data indicative of the set of environmental conditions includes at least one of: i) third party data indicative of one or more of weather conditions, time of day, or density of static objects for a portion of the environment, or ii) sensor data from at least a third sensor associated with the vehicle, wherein the third sensor includes one or more of a humidity sensor, a thermal sensor, or a vibration sensor;

determining that the set of environmental conditions meets one or more criteria for optimal calibration of the first sensor; and

initiating the calibration of the first sensor at least partially in response to determining that the set of environmental conditions satisfies the one or more criteria for optimal calibration of the first sensor.

11. The computer-implemented method of claim 1, wherein the first sensor is a light detection and ranging (LiDAR) sensor and the second sensor is an inertial sensor.

12. A system for calibrating a first sensor associated with a vehicle using a detected object, the system comprising:

at least one processor; and

at least one memory storing computer-executable instructions, wherein the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:

capturing, via the first sensor, point cloud data, at least a portion of which is associated with an object in an environment through which the vehicle is traveling;

determining that the object is a static mapping object;

determining that the static mapping object satisfies one or more criteria for calibrating the first sensor; and

calibrating the first sensor to a global coordinate system via a second sensor associated with the vehicle relative to the point cloud data, wherein the second sensor has been pre-calibrated to the global coordinate system.

13. The system of claim 12, wherein the at least one processor is configured by executing the computer-executable instructions to determine that the object is the static mapping object to:

determining a location of the object within the global coordinate system;

retrieving map data of the environment through which the vehicle is traveling; and

determining that the location of the object corresponds to a location of the static mapping object in the map data.

14. The system of claim 13, wherein the at least one processor is configured to calibrate the first sensor to the global coordinate system by executing the computer-executable instructions to:

determining a transformation matrix from a first local coordinate system of the first sensor to a second local coordinate system of the second sensor, wherein the transformation matrix comprises at least one rotational component and at least one translational component of the transformation matrix.

15. The system of claim 14, wherein the at least one rotational component and at least one translational component align at least a portion of the point cloud data with the position of the static mapping object within the global coordinate system as a relative position of the vehicle relative to the static mapping object changes.

16. The system of claim 14, wherein the at least one processor is configured to calibrate the first sensor by executing the computer-executable instructions to determine whether the transformation matrix causes the calibration accuracy of the first sensor to be within a calibration accuracy threshold.

17. The system of claim 16, wherein the transformation matrix is a first transformation matrix, and wherein the at least one processor is configured by executing the computer-executable instructions to calibrate the first sensor to the global coordinate system to:

determining a location of the object in the first local coordinate system based at least in part on at least a portion of the point cloud data associated with the object;

applying the first transformation matrix to the location of the object in the first local coordinate system to obtain a corresponding location in the second local coordinate system;

applying a second transformation matrix to corresponding locations in the second local coordinate system to obtain calibrated locations of the object in the global coordinate system; and

determining the calibration accuracy of the calibration of the first sensor based at least in part on a deviation between the calibrated location of the object in the global coordinate system obtained via applying the first and second transformation matrices and a location of the static mapping object in the map data.

18. The system of claim 17, wherein determining whether the transformation matrix causes a calibration accuracy of the first sensor to be within a calibration accuracy threshold comprises determining that the calibration accuracy of the first sensor is not within the calibration accuracy threshold, wherein at least a portion of the point cloud data associated with the object is a first portion of the point cloud data and the static mapping object is a first static mapping object, and wherein the at least one processor is further configured to execute the computer-executable instructions to:

determining that a second object in the environment through which the vehicle is traveling is a second statically mapped object, a second portion of the point cloud data being associated with the second statically mapped object;

re-determining the transformation matrix relative to the second portion of the point cloud data; and

re-determining the calibration accuracy of the first sensor based at least in part on the re-determined transformation matrix.

19. The system of claim 12, wherein the at least one processor is configured to determine, by executing the computer-executable instructions, that the static mapping object satisfies the one or more criteria for calibrating the first sensor to:

determining at least one of a shape feature, a size feature, or a surface feature of the statically mapped object;

determining a similarity metric indicative of a level of similarity between at least one of a shape feature, a size feature, or a surface feature of the statically mapped object and one or more features preselected to be optimal for calibrating the first sensor; and

determining that the similarity metric satisfies a threshold.

20. The system of claim 12, wherein the at least one processor is further configured to execute the computer-executable instructions to, prior to calibrating the first sensor:

identifying data indicative of a set of environmental conditions in the environment in which the vehicle is traveling, wherein the data indicative of the set of environmental conditions includes at least one of: i) third party data indicative of one or more of weather conditions, time of day, or density of static objects for a portion of the environment, or ii) sensor data from at least a third sensor associated with the vehicle, wherein the third sensor includes one or more of a humidity sensor, a thermal sensor, or a vibration sensor;

determining that the set of environmental conditions meets one or more criteria for optimal calibration of the first sensor; and

initiating the calibration of the first sensor at least partially in response to determining that the set of environmental conditions satisfies the one or more criteria for optimal calibration of the first sensor.