JP7572420B2 - SYSTEM AND METHOD FOR REAL-TIME CONTROL OF AUTONOMOUS DEVICES - Patent application - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
This utility patent application is a continuation-in-part of U.S. patent application Ser. No. 16/800,497 (Attorney Docket No. AA164), filed Feb. 25, 2020, and entitled "System and Method for Surface Feature Detection and Traversal," which is incorporated by reference in its entirety. This patent application is a joint venture between U.S. Provisional Patent Application No. 62/872,396 (Attorney Docket No. AA028), filed on July 10, 2019 and entitled “Apparatus for Long and Short Range Sensors on an Autonomous Delivery Device,” U.S. Provisional Patent Application No. 62/990,485 (Attorney Docket No. AA037), filed on March 17, 2020 and entitled “System and Method for Managing an Occupancy Grid,” and U.S. Provisional Patent Application No. 62/109,485 (Attorney Docket No. AA039), filed on July 10, 2019 and entitled “System and Method for Real-Time This application claims the benefit of U.S. Provisional Patent Application No. 62/872,320 (Attorney Docket No. Z96), entitled "Control of the Configuration of an Autonomous Device."

This application is incorporated herein by reference in its entirety, in accordance with U.S. Patent Application No. 16/035,205 (Attorney Docket No. X80), filed on July 13, 2018 and entitled "MOBILITY DEVICE," U.S. Patent Application No. 15/787,613 (Attorney Docket No. W10), filed on October 18, 2017 and entitled "MOBILITY DEVICE," U.S. Patent Application No. 15/600,703 (Attorney Docket No. U22), filed on May 20, 2017 and entitled "MOBILITY DEVICE," U.S. Patent Application No. 15/600,703 (Attorney Docket No. U22), filed on May 17, 2018 and entitled "SYSTEM AND METHOD FOR SECURE REMOTE CONTROL OF A ... No. 15/982,737 (Attorney Docket No. X55), entitled "MEDICAL DEVICE", U.S. Provisional Application No. 62/532,993 (Attorney Docket No. U30), entitled "MOBILITY DEVICE IMPROVEMENTS", filed July 15, 2017, U.S. Provisional Application No. 62/559,263 (Attorney Docket No. V85), entitled "MOBILITY DEVICE SEAT", filed September 15, 2017, and U.S. Provisional Application No. 62/581,670 (Attorney Docket No. W07), entitled "MOBILITY DEVICE SEAT", filed November 4, 2017.

The present teachings relate generally to AVs, and more specifically to autonomous route planning, global occupancy grid management, on-vehicle sensors, surface feature detection and traversal, and real-time vehicle configuration changes.

(background)
Navigation of AVs and semi-autonomous vehicles (AVs) typically relies on long-range sensors, including, for example, but not limited to, LIDAR, cameras, stereo cameras, and radar. Long-range sensors can sense 4-100 meters from the AV. In contrast, object avoidance and/or surface detection typically relies on short-range sensors, including, for example, but not limited to, stereo cameras, short-range radar, and ultrasonic sensors. These short-range sensors typically observe an area or volume of about 5 meters around the AV. The sensors can, for example, allow the AV to orient within its environment and navigate roads, sidewalks, obstacles, and open spaces to reach a desired destination. The sensors can also allow for visualization of people, signs, traffic signals, obstacles, and surface features.

Surface feature traversal can be a challenge because surface features, such as but not limited to, substantially discontinuous surface features (SDSFs), can be found in heterogeneous forms, which may be unique to a specific geography. However, SDSFs, such as but not limited to slopes, edges, curbs, steps, and curb-like geometries (referred to herein in a non-limiting manner as SDSFs or simply surface features), can contain some typical characteristics that may aid in their identification. Surface/road conditions and surface types can be recognized and classified, for example, by fusing multi-sensory data, which can be complex and costly. Surface features and conditions can be used to control the physical reconstruction of the AV in real time.

The sensors can be used to enable the creation of an occupancy grid that can represent the world for route planning purposes for AVs. Route planning requires a grid that identifies spaces as free, occupied, or unknown. However, the probability that a space is occupied can improve decision making regarding the space. A log-odds representation of the probability can be used to increase the precision in the numerical bounds of the 0 and 1 probabilities. The probability that a cell is occupied can depend at least on the new sensor information, the previous sensor information, and the prior occupancy information.

What is needed is a system that combines aggregated and real-time sensor data with changes in the vehicle's physical configuration to accomplish variable terrain traversal. What is needed is advantageous sensor placement to accomplish physical configuration changes, variable terrain traversal, and object avoidance. What is needed is the ability to localize SDSFs based on a multipart model associated with several criteria for SDSF identification. What is needed is the determination of candidate surface feature traversal based on criteria such as candidate traversal approach angle, candidate traversal driving surface on both sides of the candidate surface feature, and real-time determination of candidate traversal path obstacles. What is needed is a system and method for incorporating drivable surface and device mode information into occupancy grid determination.

(summary)
An AV of the present teachings can autonomously navigate to a desired location. In some configurations, the AV can include a device controller, including a sensor, a perception subsystem, an autonomy subsystem, a driver subsystem, a power base, four powered wheels, two caster wheels, and a cargo container. In some configurations, the perception and autonomy subsystem can receive and process sensor information (perception) and map information (perception and autonomy) and can provide instructions to the driver subsystem. The map information can include surface classifications and associated device modes. The movement of the AV, controlled by the driver subsystem and enabled by the power base, can be sensed by the sensor subsystem and provide a feedback loop. In some configurations, the SDSF can be accurately identified from point cloud data and stored in a map, for example, according to the processes described herein. Portions of the map associated with the location of the AV can be provided to the AV during navigation. The perception subsystem can maintain an occupancy grid that can inform the AV about the probability that the path to be traversed is currently occupied. In some configurations, the AV can operate in multiple distinct modes. Modes can enable complex terrain traversal, among other benefits: a combination of maps (e.g., surface classification), sensor data (sensing features surrounding the AV), occupancy grids (probability that an upcoming path point is occupied), and modes (whether difficult terrain can be traversed) can be used to identify the AV's direction, configuration, and speed.

Regarding preparing a map, in some configurations, a method of the present teachings for creating a map for navigating at least one SDSF encountered by an AV, wherein the AV travels a path over a surface, the surface including at least one SDSF, the path including a start point and an end point, the method can include, without limitation, accessing point cloud data representing the surface, filtering the point cloud data, forming the filtered point cloud data into a processable portion, and merging the processable portion into at least one concave polygon. The method can include locating and labeling at least one SDSF within the at least one concave polygon. The locating and labeling can form the labeled point cloud data. The method can include creating a graphing polygon based at least on the at least one concave polygon, and choosing a path from the start point to the end point based at least on the graphing polygon. When navigating, the AV can traverse at least one SDSF along the path.

Filtering the point cloud data may optionally include conditionally removing points representing transient objects and points representing outliers from the point cloud data and replacing the removed points with a preselected height. Forming the processing portion may optionally include segmenting the point cloud data into processable portions and removing points at preselected heights from the processable portions. Merging the processable portions may optionally include reducing a size of the processable portion by analyzing outliers, voxels, and normals, growing an area from the reduced-sized processable portion, determining an initial drivable surface from the grown area, segmenting and meshing the initial drivable surface, locating polygons in the segmented and meshed initial drivable surface, and setting the drivable surface based at least on the polygons. Locating and labeling the at least one SDSF feature may optionally include sorting the drivable surface point cloud data according to an SDSF filter, the SDSF filter including at least three categories of points, and locating the at least one SDSF point based at least on whether the categories of points, in combination, satisfy at least one first preselected criterion. The method may optionally include creating at least one SDSF trajectory based at least on whether a plurality of the at least one SDSF points, in combination, satisfy at least one second preselected criterion. Creating a graphing polygon may further optionally include creating at least one polygon from the at least one drivable surface. The at least one polygon may include an edge. Creating the graphed polygon may include smoothing the edges, forming a running margin based on the smoothed edges, adding at least one SDSF trajectory to the at least one drivable surface, and removing the edges from the at least one drivable surface according to at least one third preselected criterion. Smoothing the edges may optionally include trimming the edges outward. Forming a running margin of the smoothed edges may optionally include trimming the outward edges inward.

In some configurations, a system of the present teachings for creating a map for navigating at least one SDSF encountered by an AV, the AV traversing a path over a surface, the surface including at least one SDSF, the path including a start point and an end point, the system may include, without limitation, a first processor that accesses point cloud data representing the surface, a first filter that filters the point cloud data, a second processor that forms a processable portion from the filtered point cloud data, a third processor that merges the processable portion into at least one concave polygon, a fourth processor that locates and labels the at least one SDSF within the at least one concave polygon, the locating and labeling forming the labeled point cloud data, a fifth processor that creates a graphing polygon, and a path selector that selects a path from the start point to the end point based at least on the graphing polygon. The AV may traverse at least one SDSF along the path.

The first filter may optionally include executable code that may include, but is not limited to, conditionally removing points representing transient objects and points representing outliers from the point cloud data and replacing the removed points with a preselected height. The segmenter may optionally include executable code that may include, but is not limited to, segmenting the point cloud data into processable portions and removing points at preselected heights from the processable portions. The third processor may optionally include executable code that may include, but is not limited to, reducing a size of the processable portion by analyzing outliers, voxels, and normals, growing an area from the reduced-sized processable portion, determining an initial drivable surface from the grown area, segmenting and meshing the initial drivable surface, locating polygons in the segmented and meshed initial drivable surface, and setting the drivable surface based at least on the polygons. The fourth processor may optionally include executable code that may include, but is not limited to, sorting the drivable surface point cloud data according to an SDSF filter, the SDSF filter including at least three categories of points, and locating at least one SDSF point based on whether the category points, in combination, satisfy at least one first preselected criterion. The system may optionally include executable code that may include, but is not limited to, creating at least one SDSF trajectory based on whether a plurality of at least one SDSF point, in combination, satisfies at least one second preselected criterion.

Creating a graphed polygon may optionally include executable code that may include, but is not limited to, creating at least one polygon from at least one drivable surface, the at least one polygon including an edge, smoothing the edge, forming a running margin based on the smoothed edge, adding at least one SDSF trajectory to the at least one drivable surface, and removing the edge from the at least one drivable surface according to at least one third preselected criterion. Smoothing the edge may optionally include executable code that may include, but is not limited to, trimming the edge outward. Forming a running margin of the smoothed edge may optionally include executable code that may include, but is not limited to, trimming the outward edge inward.

In some configurations, a method of the present teachings for creating a map for navigating at least one SDSF encountered by an AV, the AV navigating a path over a surface, the surface including at least one SDSF, the path including a start point and an end point, the method can include, but is not limited to, accessing a route configuration. The route configuration can include at least one graphed polygon that can include filtered point cloud data. The point cloud data can include labeled features and a drivable margin. The method can include transforming the point cloud data to a global coordinate system, determining a boundary of the at least one SDSF, creating an SDSF buffer of a preselected size around the boundary, determining at least one SDSF that can be traversed based at least on at least one SDSF traversal criterion, creating an edge/weighted graph based at least on the at least one SDSF traversal criterion, the transformed point cloud data, and the route configuration, and selecting a path from the start point to a destination point based at least on the edge/weighted graph.

The at least one SDSF crossing criterion may optionally include a preselected width of the at least one SDSF and a preselected smoothness of the at least one SDSF, a minimum entry and exit distance between the at least one SDSF and the AV that includes a drivable surface, and a minimum entry distance between the at least one SDSF and the AV that can accommodate an approximately 90° approach to the at least one SDSF by the AV.

In some configurations, a system of the present teachings for creating a map for navigating at least one SDSF encountered by an AV, where the AV travels a path over a surface, the surface including at least one SDSF, the path including a start point and an end point, the system may include, but is not limited to, a sixth processor that accesses a route configuration. The route configuration may include at least one graphed polygon that may include filtered point cloud data. The point cloud data may include labeled features and a drivable margin. The system may include a seventh processor that transforms the point cloud data to a global coordinate system and an eighth processor that determines a boundary of the at least one SDSF. The eighth processor may create an SDSF buffer of a preselected size around the boundary. The system may include a ninth processor that determines at least one SDSF that may be traversed based on at least one SDSF traversal criterion, a tenth processor that creates an edge/weighted graph based on at least one SDSF traversal criterion, the transformed point cloud data, and the route morphology, and a base controller that selects a route from a start point to a destination point based on at least the edge/weighted graph.

In some configurations, a method of the present teachings for creating a map for navigating at least one SDSF encountered by an AV, where the AV travels a path over a surface, the surface including at least one SDSF, the path including a start point and an end point, the method can include, without limitation, accessing point cloud data representing the surface. The method can include filtering the point cloud data, forming the filtered point cloud data into a processable portion, and merging the processable portion into at least one concave polygon. The method can include locating and labeling at least one SDSF within the at least one concave polygon. The locating and labeling can form the labeled point cloud data. The method can include creating a graphing polygon based at least on the at least one concave polygon. The graphing polygon can form a route form, and the point cloud data can include the labeled features and the driveable margin. The method may include transforming the point cloud data into a global coordinate system, determining a boundary of at least one SDSF, creating an SDSF buffer of a preselected size around the boundary, determining at least one SDSF that may be traversed based at least on at least one SDSF traversal criterion, creating an edge/weighted graph based at least on the at least one SDSF traversal criterion, the transformed point cloud data, and a route morphology, and selecting a route from a start point to a destination point based at least on the edge/weighted graph.

Filtering the point cloud data may optionally include conditionally removing points representing transient objects and points representing outliers from the point cloud data and replacing the removed points with a preselected height. Forming the processing portion may optionally include segmenting the point cloud data into processable portions and removing points at preselected heights from the processable portions. Merging the processable portions may optionally include reducing a size of the processable portion by analyzing outliers, voxels, and normals, growing an area from the reduced-sized processable portion, determining an initial drivable surface from the grown area, segmenting and meshing the initial drivable surface, locating polygons in the segmented and meshed initial drivable surface, and setting the drivable surface based at least on the polygons. Locating and labeling the at least one SDSF feature may optionally include sorting the drivable surface point cloud data according to an SDSF filter, the SDSF filter including at least three categories of points, and locating the at least one SDSF point based at least on whether the categories of points, in combination, satisfy at least one first preselected criterion. The method may optionally include creating at least one SDSF trajectory based at least on whether a plurality of the at least one SDSF points, in combination, satisfy at least one second preselected criterion. Creating a graphing polygon may further optionally include creating at least one polygon from the at least one drivable surface. The at least one polygon may include an edge. Creating the graphed polygon may include smoothing the edges, forming a running margin based on the smoothed edges, adding at least one SDSF trajectory to the at least one drivable surface, and removing the edges from the at least one drivable surface according to at least one third preselected criterion. Smoothing the edges may optionally include trimming the edges outward. Forming a running margin of the smoothed edges may optionally include trimming the outward edges inward. The at least one SDSF crossing criterion may optionally include a preselected width of the at least one SDSF and a preselected smoothness of the at least one SDSF, a minimum entry distance and a minimum exit distance between the at least one SDSF and the AV that includes the drivable surface, and a minimum entry distance between the at least one SDSF and the AV that may accommodate an approximately 90° approach by the AV to the at least one SDSF.

In some configurations, a system of the present teachings for creating a map for navigating at least one SDSF encountered by an AV, the AV navigating a path over a surface, the surface including at least one SDSF, the path including a start point and an end point, the system may include, without limitation, a point cloud accessor that accesses point cloud data representing the surface, a first filter that filters the point cloud data, a segmenter that forms a processable portion from the filtered point cloud data, a third processor that merges the processable portion into at least one concave polygon, a fourth processor that locates and labels the at least one SDSF within the at least one concave polygon, the locating and labeling forming a labeled point cloud data, and a fifth processor that creates a graphing polygon. The route form may include at least one graphing polygon that may include the filtered point cloud data. The point cloud data may include a labeled feature and a driveable margin. The system may include a seventh processor that transforms the point cloud data to a global coordinate system, and an eighth processor that determines a boundary of at least one SDSF. The eighth processor may create an SDSF buffer of a preselected size around the boundary. The system may include a ninth processor that determines at least one SDSF that may be traversed based on at least one SDSF traversal criterion, a tenth processor that creates an edge/weight graph based on at least the at least one SDSF traversal criterion, the transformed point cloud data, and a route morphology, and a base controller that plans a route from a start point to a destination point based on at least the edge/weight graph.

The first filter may optionally include executable code that may include, but is not limited to, conditionally removing points representing transient objects and points representing outliers from the point cloud data and replacing the removed points with a preselected height. The segmenter may optionally include executable code that may include, but is not limited to, segmenting the point cloud data into processable portions and removing points at preselected heights from the processable portions. The third processor may optionally include executable code that may include, but is not limited to, reducing a size of the processable portion by analyzing outliers, voxels, and normals, growing an area from the reduced-sized processable portion, determining an initial drivable surface from the grown area, segmenting and meshing the initial drivable surface, locating polygons in the segmented and meshed initial drivable surface, and setting the drivable surface based at least on the polygons. The fourth processor may optionally include executable code that may include, but is not limited to, sorting the drivable surface point cloud data according to an SDSF filter, the SDSF filter including at least three categories of points, and locating at least one SDSF point based on whether the category points, in combination, satisfy at least one first preselected criterion. The system may optionally include executable code that may include, but is not limited to, creating at least one SDSF trajectory based on whether a plurality of at least one SDSF point, in combination, satisfies at least one second preselected criterion.

Creating a graphed polygon may optionally include executable code that may include, but is not limited to, creating at least one polygon from at least one drivable surface, the at least one polygon including an edge, smoothing the edge, forming a running margin based on the smoothed edge, adding at least one SDSF trajectory to the at least one drivable surface, and removing the edge from the at least one drivable surface according to at least one third preselected criterion. Smoothing the edge may optionally include executable code that may include, but is not limited to, trimming the edge outward. Forming a running margin of the smoothed edge may optionally include executable code that may include, but is not limited to, trimming the outward edge inward.

In some configurations, the SDSF can be identified by its dimensions. For example, a curb may include, but is not limited to, a width of approximately 0.6-0.7 m. In some configurations, point cloud data can be processed to locate the SDSF, which can be used to prepare a route for the AV from a start point to a destination. In some configurations, the route can be included in a map and provided to a perception subsystem. As the AV progresses along the route, in some configurations, SDSF traversal can be adapted, in part, through sensor-based positioning of the AV, enabled by the perception subsystem. The perception subsystem can be executed on at least one processor within the AV.

The AV may include a power base, including, but not limited to, two powered front wheels, two powered rear wheels, and an energy storage device, and at least one processor. The power base may be configured to move at a commanded speed. The AV may include a cargo platform mechanically attached to the power base and including a plurality of short-range sensors. The AV may include a cargo container, in some configurations, mounted on the cargo platform and having a volume for receiving one or more objects to be delivered. The AV may include a long-range sensor suite, in some configurations, mounted on the cargo container and may include, but not limited to, a LIDAR and one or more cameras. The AV may include a controller, which may receive data from the long-range and short-range sensor suites.

The short-range sensor suite can optionally detect at least one characteristic of the drivable surface and can optionally include a stereo camera, an IR projector, two image sensors, an RGB sensor, and a radar sensor. The short-range sensor suite can optionally provide RGB-D data to the controller. The controller can optionally determine the geometry of the road surface based on the RGB-D data received from the short-range sensor suite. The short-range sensor suite can optionally detect objects within 4 meters of the AV, and the long-range sensor suite can optionally detect objects greater than 4 meters from the AV.

The perception subsystem can use the data collected by the sensors to populate the occupancy grid. The occupancy grid of the present teachings is configured as a 3D grid of points surrounding the AV, where the AV can occupy a center point. In some configurations, the occupancy grid can extend 10 m to the left, right, rear, and front of the AV. The grid can include approximately the height of the AV and virtually travel with the AV as it moves, representing obstacles surrounding the AV. The grid can be converted to two dimensions by shrinking its vertical axis, for example, but not by way of limitation, divided into polygons approximately 5 cm by 5 cm in size. Obstacles that appear in the 3D space around the AV can be reduced to 2D shapes. If the 2D shape overlaps any segment of one of the polygons, the polygon can be given a value of 100, indicating that the space is occupied. Any polygons that are left unfilled can be given a value of 0 and referred to as free space into which the AV may move.

As an AV navigates, it may encounter situations that may require a change in the AV's configuration. A method of the present teachings for real-time control of a device's configuration, in some configurations, the device includes a chassis, at least four wheels, a first side of the chassis operably coupled to at least one of the at least four wheels, and an opposing second side of the chassis operably coupled to at least one of the at least four wheels, the method may include, without limitation, receiving environmental data, determining a surface type based at least on the environmental data, determining a mode based at least on the surface type and the first configuration, determining a second configuration based at least on the mode and surface type, determining a movement command based on at least the second configuration, and controlling the configuration of the device by using the movement command to change the device from the first configuration to the second configuration.

The method may optionally include capturing an occupancy grid based on at least a surface type and a mode. The environmental data may optionally include RGB-D image data and a morphology of a road surface. The configuration may optionally include two clustered pairs of at least four wheels. A first pair of the two pairs may be positioned on a first side and a second pair of the two pairs may be positioned on a second side. The first pair may include a first front wheel and a first rear wheel, and the second pair may include a second front wheel and a second rear wheel. Control of the configuration may optionally include coordinated powering of the first pair and the second pair based at least on the environmental data. Control of the configuration may optionally include transitioning from driving the at least four wheels and a pair of casters that are retracted to driving two wheels with the clustered first pair and the clustered second pair rotated to lift the first front wheel and the second front wheel. The pair of casters can be operably coupled to the chassis. The device can be resting on the first rear wheel, the second rear wheel, and the pair of casters. Controlling the configuration can optionally include rotating the pair of clusters operably coupled with the two powered wheels on the first side and the two powered wheels on the second side based on at least the environmental data.

A system of the present teachings for real-time control of a configuration of an AV can include, but is not limited to, a device processor and a power base processor. The AV can include a chassis, at least four wheels, a first side of the chassis, and an opposing second side of the chassis. The device processor can receive real-time environmental data surrounding the AV, determine a surface type based on at least the environmental data, determine a mode based on at least the surface type and the first configuration, and determine a second configuration based on at least the mode and surface type. The power base processor can enable the AV to move based on at least the second configuration and can enable the AV to change from the first configuration to the second configuration. The device processor can optionally include populating an occupancy grid based on at least the surface type and the mode.

During navigation, the AV may encounter SDSFs that may require the AV to steer for successful traversal. In some configurations, a method of the present teachings for navigating an AV along a path line within a going area toward a target point that traverses at least one SDSF, the AV including a leading edge and a trailing edge, the method may include, without limitation, receiving SDSF information and obstacle information regarding the going area, detecting at least one candidate SDSF from the SDSF information, and selecting an SDSF line from the at least one candidate SDSF line based on at least one selection criterion. The method may include determining at least one traversable portion of the selected SDSF line based on at least one location of at least one obstacle found in the obstacle information in the vicinity of the selected SDSF line, orienting the AV toward the at least one traversable portion by turning the AV to travel along a line perpendicular to the traversable portion and operating at a first speed, and constantly correcting the AV's orientation based on a relationship between the orientation and the perpendicular line. The method may include running the AV at a second speed by adjusting the first speed of the AV based on at least the orientation and the distance between the AV and the traversable portion. If the SDSF associated with the at least one traversable portion is elevated relative to the surface of the travel route, the method may include crossing the SDSF by elevating the leading edge relative to the trailing edge and running the AV at a third increased speed per degree of elevation, and running the AV at a fourth speed until the AV passes the SDSF.

Detecting at least one candidate SDSF from the SDSF information may optionally include: (a) drawing a closed polygon encompassing the AV location and the target point location; (b) drawing a path line between the target point and the AV location; (c) selecting two SDSF points from the SDSF information, the SDSF points being located within the polygon; and (d) drawing an SDSF line between the two points. Detecting at least one candidate SDSF may include (e) repeating steps (c)-(e) if there are less than a first preselected number of points within a first preselected distance of the SDSF line, and if there are less than a second preselected number of attempts in selecting SDSF points, drawing a line between them, and having less than the first preselected number of points around the SDSF line. Detecting at least one candidate SDSF may include (f) fitting a curve to the SDSF points that fall within a first preselected distance of the SDSF line if a first preselected number of points or more exists, and (g) identifying the curve as an SDSF line if a first number of SDSF points within the first preselected distance of the curve exceeds a second number of SDSF points within the first preselected distance of the SDSF line, and if the curve intersects with a path line, and if there are no gaps between the SDSF points on the curve that exceed the second preselected distance. Detecting at least one candidate SDSF may include (h) repeating steps (f)-(h) if the number of points within a first preselected distance of the curve does not exceed the number of points within a first preselected distance of the SDSF line, or if the curve does not intersect with the path line, or if there are gaps between SDSF points on the curve that exceed a second preselected distance, and if the SDSF line does not remain stable, and if steps (f)-(h) have not been attempted more than a second preselected number of attempts.

The closed polygon may optionally include a preselected width, which may optionally include a width dimension of the AV. Selecting the SDSF points may optionally include random selection. At least one selection criterion may optionally include a first number of SDSF points within a first preselected distance of the curve exceeding a second number of SDSF points within a first preselected distance of the SDSF line, the curve intersecting the path line, and no gaps between the SDSF points on the curve exceeding a second preselected distance.

Determining at least one traversable portion of the selected SDSF can optionally include selecting a plurality of obstacle points from the obstacle information. Each of the plurality of obstacle points can include a probability that the obstacle point is associated with at least one obstacle. Determining at least one traversable portion can include projecting the plurality of obstacle points onto the SDSF line to form at least one projection if the probability is higher than a preselected percentage and if any of the plurality of obstacle points is located between the SDSF line and the target point and if any of the plurality of obstacle points is less than a third preselected distance from the SDSF line. Determining at least one traversable portion can optionally include connecting at least two of the at least one projection to each other, locating end points of the at least two connected projections along the SDSF line, marking the at least two connected projections as non-traversable SDSF segments, and marking the SDSF line outside the non-traversable segments as at least one traversable segment.

Traversing at least one traversable portion of the SDSF can optionally include turning the AV to travel along a line perpendicular to the traversable portion, orienting the AV toward the traversable portion and operating at a first speed, constantly correcting the orientation of the AV based on a relationship between the orientation and the perpendicular line, and running the AV at a second speed by adjusting the first speed of the AV based on at least the orientation and the distance between the AV and the traversable portion. Traversing at least one traversable portion of the SDSF can optionally include traversing the SDSF by elevating the leading edge relative to the trailing edge and running the AV at a third increased speed per degree of elevation if the SDSF rises relative to the surface of the travel route, and running the AV at a fourth speed until the AV passes the SDSF.

Traversing at least one traversable portion of the SDSF may alternatively and optionally include: (a) ignoring updates to the SDSF information and running the AV at a preselected speed if the heading error is below a third preselected amount relative to a line perpendicular to the SDSF line; (b) running the AV forward and increasing the speed of the AV to an eighth preselected speed per degree of rise if the elevation of the front portion of the AV relative to the rear portion of the AV is between a sixth preselected amount and a fifth preselected amount; (c) running the AV forward at a seventh preselected speed if the front portion is elevated below the sixth preselected amount relative to the rear portion; and (d) repeating steps (a)-(d) if the rear portion is less than or equal to a fifth preselected distance from the SDSF line.

In some configurations, the SDSF and AV wheels can be automatically aligned to avoid system instability. Automatic alignment can be implemented, for example, but not limited to, by continuously testing and correcting the orientation of the AV as it approaches the SDSF. Another aspect of the SDSF traversal feature of the present teachings is that it automatically verifies that sufficient free space exists around the SDSF before attempting to cross. Yet another aspect of the SDSF traversal feature of the present teachings is that it is capable of traversing SDSFs of various geometries. The geometries can include, for example, but not limited to, square and contoured SDSFs. The orientation of the AV relative to the SDSF can determine the speed and direction in which the AV travels. The SDSF traversal feature can adjust the speed of the AV in the vicinity of the SDSF. When the AV rises into the SDSF, the speed can be increased to assist the AV in crossing the SDSF.

1. An autonomous delivery vehicle comprising: a power base including two powered front wheels, two powered rear wheels, and an energy storage device, the power base configured to move at a commanded speed and in a commanded direction to perform a transport of at least one object; a cargo platform including a plurality of short-range sensors, the cargo platform mechanically attached to the power base; a cargo container with a volume for receiving at least one object, the cargo container mounted on the cargo platform; a long-range sensor suite including a LIDAR and one or more cameras, the long-range sensor suite mounted on the cargo container; and a controller for receiving data from the long-range sensor suite and the plurality of short-range sensors, the controller determining a commanded speed and a commanded direction based at least on the data, and the controller providing the commanded speed and commanded direction to the power base to complete the transport. 2. 3. The autonomous delivery vehicle of claim 1, wherein the data from the plurality of short-range sensors comprises at least one characteristic of a surface over which the power base travels. 4. The autonomous delivery vehicle of claim 1, wherein the plurality of short-range sensors comprises at least one IR projector, at least one image sensor, and at least one RGB sensor. 5. The autonomous delivery vehicle of claim 1, wherein the plurality of short-range sensors comprises at least one radar sensor. 6. The autonomous delivery vehicle of claim 1, wherein the data from the plurality of short-range sensors comprises RGB-D data. 7. The autonomous delivery vehicle of claim 1, wherein the controller determines a geometry of a road surface based on the RGB-D data received from the plurality of short-range sensors. 8. The autonomous delivery vehicle of claim 1, wherein the plurality of short-range sensors detect objects within 4 meters of the AV and the long-range sensor suite detects objects greater than 4 meters from the autonomous delivery vehicle. 9. Item 10. The autonomous delivery vehicle of item 1, wherein the plurality of short-range sensors comprises a cooling circuit. Item 11. The autonomous delivery vehicle of item 1, wherein the plurality of short-range sensors comprises an ultrasonic sensor. Item 3. The autonomous delivery vehicle of item 2, wherein the controller comprises executable code for: accessing a map, the map formed by a map processor, the map processor accessing point cloud data from the long range sensor suite, the point cloud data representing a surface; a first processor for filtering the point cloud data, the filter and forming a processable portion from the filtered point cloud data; a third processor for merging the processable portion into at least one polygon; a fourth processor for locating and labeling at least one substantially discontinuous surface feature (SDSF) within the at least one polygon, if present, the locating and labeling forming labeled point cloud data; a fifth processor for creating a graphing polygon from the labeled point cloud data; and a sixth processor for planning a route from a start point to an end point based at least on the graphing polygon, wherein the AV traverses the at least one SDSF along the route. 12. The autonomous delivery vehicle of claim 11, wherein the filter comprises a seventh processor executing code including: conditionally removing points representing transient objects and points representing outliers from the point cloud data, and replacing the removed points with a preselected height. 13. The autonomous delivery vehicle of claim 11, wherein the second processor includes executable code including: segmenting the point cloud data into processable portions, and removing points of preselected heights from the processable portions. 14. The autonomous delivery vehicle of claim 11, wherein the third processor includes executable code including: reducing a size of the processable portion by analyzing outliers, voxels, and normals, growing an area from the reduced-sized processable portion, determining an initial drivable surface from the grown-up area, segmenting and meshing the initial drivable surface, locating polygons in the segmented and meshed initial drivable surface, and setting at least one drivable surface based at least on the polygons. 15. 16. The autonomous delivery vehicle of claim 14, wherein the fourth processor includes executable code including: sorting the initial drivable surface point cloud data according to an SDSF filter, the SDSF filter including at least three categories of points; and locating at least one SDSF point based on whether the at least three categories of points, in combination, satisfy at least one first preselected criterion. 17. The autonomous delivery vehicle of claim 15, wherein the fourth processor includes executable code including: creating at least one SDSF trajectory based on whether a plurality of at least one SDSF point, in combination, satisfy at least one second preselected criterion. Item 14. The autonomous delivery vehicle of item 14, including an eighth processor including executable code for: creating at least one polygon from the at least one drivable surface, the at least one polygon including an outer edge; smoothing the outer edge; forming a driving margin based on the smoothed outer edge; adding at least one SDSF trajectory to the at least one drivable surface; and removing an inner edge from the at least one drivable surface according to at least one third preselected criterion. 18. The autonomous delivery vehicle of item 17, including a ninth processor including executable code for: smoothing the outer edge includes trimming the outer edge outward to form an outward edge. 19. The autonomous delivery vehicle of item 18, including a tenth processor including executable code for: forming a driving margin of the smoothed outer edge includes trimming the outward edge inward. 20. The controller includes a subsystem for navigating at least one substantially discontinuous surface feature (SDSF) encountered by an autonomous delivery vehicle (AV), where the AV travels a path over the surface, the surface including at least one SDSF, the path including a start point and an end point, a first processor that accesses a route configuration, the route configuration including at least one graphed polygon including filtered point cloud data, the filtered point cloud data including the labeled features, and the point cloud data including a drivable margin, a second processor that transforms the point cloud data into a global coordinate system, and at least 20. The autonomous delivery vehicle of item 1, comprising a subsystem comprising: a third processor that determines a boundary of at least one SDSF, the third processor creating an SDSF buffer of a preselected size around the boundary; a fourth processor that determines at least one SDSF that may be traversed based on at least one SDSF traversal criterion; a fifth processor that creates an edge/weighted graph based on at least the at least one SDSF traversal criterion, the transformed point cloud data, and the route morphology; and a base controller that plans a route from a start point to an end point based on at least the edge/weighted graph. The autonomous delivery vehicle of item 20, wherein the at least one SDSF crossing criterion comprises a preselected width of the at least one SDSF and a preselected smoothness of the at least one SDSF, a minimum entry distance and a minimum exit distance between the at least one SDSF and the AV, including a drivable surface, and a minimum entry distance between the at least one SDSF and the AV that accommodates an approximately 90° approach to the at least one SDSF by the AV.

22. A method for managing a global occupancy grid for an autonomous device, the global occupancy grid including global occupancy grid cells, the global occupancy grid cells being associated with occupancy probabilities, comprising: receiving sensor data from a sensor associated with the autonomous device; creating a local occupancy grid based on at least the sensor data, the local occupancy grid having local occupancy grid cells; when the autonomous device moves from a first area to a second area, accessing historical data associated with the second area; creating a static grid based on at least the historical data; and moving the global occupancy grid to maintain the autonomous device in a center position of the global occupancy grid. 23. The method of claim 22, further comprising: maintaining a local occupancy grid of at least one of the global occupancy grid cells based on a static grid; updating the moved global occupancy grid based on the static grid; marking at least one of the global occupancy grid cells as unoccupied if at least one of the global occupancy grid cells matches a location of the autonomous device; for each local occupancy grid cell, calculating a position of the local occupancy grid cell on the global occupancy grid; accessing a first occupancy probability from the global occupancy grid cell at the position; accessing a second occupancy probability from the local occupancy grid cell at the position; and calculating a new occupancy probability at the position on the global occupancy grid based on at least the first occupancy probability and the second occupancy probability. 24. The method of claim 23, further comprising: setting the new occupancy probability to 0 if the new occupancy probability is <0; and setting the new occupancy probability to 1 if the new occupancy probability is >1. 25. The method of claim 22, further comprising setting the global occupancy grid cell to the new occupancy probability. 26. 24. The method of claim 23, further comprising setting the global occupancy grid cells to the range-checked new occupancy probability.

27. A method for creating and managing an occupancy grid, comprising: transforming sensor measurements by a local occupancy grid creation node into a reference frame associated with the device; creating a time-stamped measurement occupancy grid; publishing the time-stamped measurement occupancy grid as a local occupancy grid; creating a plurality of local occupancy grids; creating a static occupancy grid based on surface characteristics in a repository, the surface characteristics being associated with a location of the device; moving a global occupancy grid associated with the location of the device to keep the device and the local occupancy grid approximately centered with respect to the global occupancy grid; 27. The method of claim 26, further comprising: adding information from a local occupancy grid to a global occupancy grid; marking an area in the global occupancy grid currently occupied by the device as unoccupied; and for at least one cell in each local occupancy grid, determining a location of the at least one cell in the global occupancy grid, accessing a first value at the location, determining a second value at the location based on a relationship between the first value and a cell value in the at least one cell in the local occupancy grid, comparing the second value against a preselected probability range, and setting the global occupancy grid with the new value if the probability value is within the preselected probability range. 28. The method of claim 27, further comprising: publishing the global occupancy grid. 29. The method of claim 27, wherein the surface characteristics comprise a surface type and a surface discontinuity. 30. The method of claim 27, wherein the relationship comprises summing. 31. A system for creating and managing occupancy grids, comprising: a plurality of local grid creation nodes that create at least one local occupancy grid, the at least one local occupancy grid being associated with a location of a device, the at least one local occupancy grid including at least one cell; and a global occupancy grid manager that accesses the at least one local occupancy grid, the global occupancy grid being associated with the location of the device, the global occupancy grid being moved to a location of the device and the at least one local occupancy grid being approximately centered relative to the global occupancy grid, the global occupancy grid manager accessing the at least one local occupancy grid and creating a static occupancy grid based on surface characteristics in a repository, the surface characteristics being associated with the location of the device, the global occupancy grid being moved to a location of the device and the at least one local occupancy grid being approximately centered relative to the global occupancy grid. and a global occupancy grid manager that maintains a local occupancy grid, adds information from the static occupancy grid to at least one global occupancy grid, marks areas in the global occupancy grid that are currently occupied by the device as unoccupied, and for at least one cell in each local occupancy grid, determines a location of the at least one cell in the global occupancy grid, accesses a first value at the location, determines a second value at the location based on a relationship between the first value and a cell value in the at least one cell in the local occupancy grid, compares the second value against a preselected probability range, and sets the global occupancy grid with the new value if the probability value is within the preselected probability range.

32. A method for updating a global occupancy grid, the method including: when an autonomous device moves to a new location, updating the global occupancy grid with information from a static grid associated with the new location; analyzing a surface at the new location; if the surface is drivable, updating the surface and updating the global occupancy grid with the updated surface; and updating the global occupancy grid with values from a repository of static values, the static values associated with the new location. 33. 33. The method of claim 32, wherein updating the surface includes accessing a local occupancy grid associated with the new location, for each cell in the local occupancy grid, accessing a local occupancy grid surface classification confidence value and a local occupancy grid surface classification, if the local occupancy grid surface classification is identical to the global surface classification in the global occupancy grid in the cell, adding the global surface classification confidence value in the global occupancy grid to the local occupancy grid surface classification confidence value to form a sum and updating the global occupancy grid at the cell using the sum, if the local occupancy grid surface classification is not identical to the global surface classification in the global occupancy grid in the cell, subtracting the local occupancy grid surface classification confidence value from the global surface classification confidence value in the global occupancy grid to form a difference and updating the global occupancy grid using the difference, and if the difference is less than zero, updating the global occupancy grid with the local occupancy grid surface classification. Item 33. The method of item 32, wherein updating the global occupancy grid with values from the repository of static values includes: for each cell in the local occupancy grid, accessing a log-odds value, which is a local occupancy grid probability that the cell from the local occupancy grid is an occupancy value; updating the log-odds value in the global occupancy grid with the local occupancy grid log-odds value at the cell; decreasing the log-odds that the cell is occupied in the local occupancy grid if a preselected certainty that the cell is not occupied is met and if the autonomous device is proceeding within a lane barrier and if the local occupancy grid surface classification indicates a drivable surface; increasing the log-odds in the local occupancy grid if the autonomous device expects to encounter a relatively uniform surface and if the local occupancy grid surface classification indicates a relatively non-uniform surface; and decreasing the log-odds in the local occupancy grid if the autonomous device expects to encounter a relatively uniform surface and if the local occupancy grid surface classification indicates a relatively uniform surface.

35. A method for real-time control of a configuration of a device, the device including a chassis, at least four wheels, a first side of the chassis operatively coupled to at least one of the at least four wheels, and an opposing second side of the chassis operatively coupled to at least one of the at least four wheels, the method including: creating a map based on at least a priori surface features and an occupancy grid, the map being created in non-real-time, the map including at least one location, the at least one location associated with at least one surface feature, the at least one surface feature associated with at least one surface classification and at least one mode; determining current surface features as the device progresses; updating the occupancy grid in real-time with the current surface features; and determining from the occupancy grid and the map a path the device may progress to traverse the at least one surface feature.

36. A method for real-time control of a configuration of a device, the device including a chassis, at least four wheels, a first side of the chassis operably coupled to at least one of the at least four wheels, and an opposing second side of the chassis operably coupled to at least one of the at least four wheels, the method including receiving environmental data, determining a surface type based at least on the environmental data, determining a mode based at least on the surface type and the first configuration, determining a second configuration based at least on the mode and the surface type, determining a movement command based at least on the second configuration, and controlling a configuration of the device by using the movement command to change the device from the first configuration to the second configuration. 37. The method of claim 36, wherein the environmental data comprises RGB-D image data. 38. The method of claim 36, further comprising capturing an occupancy grid based at least on the surface type and the mode, and determining the movement command based at least on the occupancy grid. 39. The method of claim 38, wherein the occupancy grid comprises at least information based on data from at least one image sensor. 40. 41. The method of claim 36, wherein the environmental data comprises a morphology of a road surface. 42. The method of claim 36, wherein the configuration comprises two clustered pairs of at least four wheels, a first pair of the two pairs being positioned on a first side and a second pair of the two pairs being positioned on a second side, the first pair including a first front wheel and a first rear wheel, and the second pair including a second front wheel and a second rear wheel. 43. The method of claim 41, wherein the control of the configuration includes coordinated powering of the first pair and the second pair based at least on the environmental data. 43. The method of claim 41, wherein the control of the configuration includes transitioning from driving at least four wheels and a pair of casters retracted, the pair of casters operatively coupled to the chassis, to driving two wheels with a clustered first pair and a clustered second pair rotated to lift a first front wheel and a second front wheel, the device resting on a first rear wheel, a second rear wheel, and a pair of casters. 44. The method of claim 41, wherein the control of the configuration includes rotating a pair of clusters operatively coupled with a first two powered wheels on a first side and a second two powered wheels on a second side based at least on the environmental data. 45. The method of claim 36, wherein the device further comprises a cargo container, the cargo container mounted on a chassis, and the chassis controls a height of the cargo container. 46. The method of claim 45, wherein the height of the cargo container is based at least on the environmental data.

47. A system for real-time control of a configuration of a device, the device including a chassis, at least four wheels, a first side of the chassis, and an opposing second side of the chassis, the device processor receiving real-time environmental data surrounding the device, the device processor determining a surface type based at least on the environmental data, the device processor determining a mode based at least on the surface type and the first configuration, the device processor determining a second configuration based at least on the mode and the surface type, and a power base processor determining movement commands based at least on the second configuration, the power base processor controlling the configuration of the device by using the movement commands to change the device from the first configuration to the second configuration. 48. The system of claim 47, the environmental data comprising RGB-D image data. 49. The system of claim 47, the device processor including populating an occupancy grid based at least on the surface type and the mode. 50. The system of claim 49, the power base processor including determining movement commands based at least on the occupancy grid. 51. The system of claim 49, wherein the occupancy grid comprises at least information based on data from at least one image sensor. 52. The system of claim 47, wherein the environmental data comprises a morphology of a road surface. 53. The system of claim 47, wherein the configuration comprises two clustered pairs of at least four wheels, a first pair of the two pairs being positioned on a first side and a second pair of the two pairs being positioned on a second side, the first pair having a first front wheel and a first rear wheel, and the second pair having a second front wheel and a second rear wheel. 54. The system of claim 53, wherein the control of the configuration includes coordinated powering of the first pair and the second pair based at least on the environmental data. 55. The system described in item 53, wherein the control of the configuration includes transitioning from driving at least four wheels and a pair of casters that are retracted, the pair of casters being operably coupled to the chassis, to driving two wheels with a clustered first pair and a clustered second pair that are rotated to lift the first front wheel and the second front wheel, the device resting on the first rear wheel, the second rear wheel, and the pair of casters.

56. A method for maintaining a global occupancy grid, comprising: locating a first position of an autonomous device; when the autonomous device moves to a second position, the second position is associated with a global occupancy grid and a local occupancy grid; updating the global occupancy grid with at least one occupancy probability value associated with the first position; updating the global occupancy grid with at least one drivable surface associated with the local occupancy grid; updating the global occupancy grid with a surface confidence associated with the at least one drivable surface; and updating the global occupancy grid with log-odds of the at least one occupancy probability value using a first Bayes function. 56. The method of claim 55, further comprising: updating a global occupancy grid with at least one drivable surface associated with the local occupancy grid when the autonomous device remains at the first location and the global occupancy grid and the local occupancy grid are co-located; updating the global occupancy grid with a surface confidence associated with the at least one drivable surface; updating the global occupancy grid with the log-odds of the at least one occupancy probability value using a second Bayes function; and adjusting the log-odds based on a characteristic associated with the at least second location. 57. The method of claim 35, wherein creating the map includes accessing point cloud data representing the surface, filtering the point cloud data, forming the filtered point cloud data into a processable portion, merging the processable portion into at least one concave polygon, locating and labeling at least one SDSF within the at least one concave polygon, where the locating and labeling forms the labeled point cloud data, creating a graphing polygon based at least on the at least one concave polygon, and plotting a path from a start point to an end point based at least on the graphing polygon, where the AV traverses the at least one SDSF along the path. 58. The method of claim 57, wherein filtering the point cloud data includes conditionally removing points representing transient objects and points representing outliers from the point cloud data, and replacing the removed points with a preselected height. Item 57, wherein forming the processable portions includes segmenting the point cloud data into processable portions and removing points at a preselected height from the processable portions. 60. The method of item 57, wherein merging the processable portions includes reducing a size of the processable portions by analyzing outliers, voxels, and normals, growing an area from the reduced-sized processable portions, determining an initial drivable surface from the grown area, segmenting and meshing the initial drivable surface, locating polygons in the segmented and meshed initial drivable surface, and setting at least one drivable surface based on at least the polygons. 61. 61. The method of claim 60, wherein locating and labeling at least one SDSF includes sorting the point cloud data of the initial drivable surface according to an SDSF filter, the SDSF filter including at least three categories of points, and locating at least one SDSF point based on whether at least the at least three categories of points, in combination, satisfy at least one first preselected criterion. 62. The method of claim 61, further including creating at least one SDSF trajectory based on at least whether a plurality of at least one SDSF point, in combination, satisfies at least one second preselected criterion. Item 62, wherein creating the graphed polygon further comprises: creating at least one polygon from at least one drivable surface, the at least one polygon including an outer edge; smoothing the outer edge; forming a running margin based on the smoothed outer edge; adding at least one SDSF trajectory to the at least one drivable surface; and removing an inner edge from the at least one drivable surface according to at least one third preselected criterion. 64. The method of item 63, wherein smoothing the outer edge comprises trimming the outer edge outward to form an outward edge. 65. The method of item 63, wherein forming a running margin of the smoothed outer edge comprises trimming the outward edge inward.

66. An autonomous delivery vehicle comprising: a power base including two powered front wheels, two powered rear wheels, and an energy storage device, the power base configured to move at a commanded speed; a cargo platform including a plurality of short-range sensors, the cargo platform mechanically attached to the power base; a cargo container with a volume for receiving one or more objects to be delivered, the cargo container mounted on the cargo platform; a long-range sensor suite including a LIDAR and one or more cameras, the long-range sensor suite being mounted on the cargo container; and a controller for receiving data from the long-range sensor suite and the plurality of short-range sensors. 67. The autonomous delivery vehicle of claim 66, wherein the plurality of short-range sensors detect at least one characteristic of a drivable surface. 68. The autonomous delivery vehicle of claim 66, wherein the plurality of short-range sensors are stereo cameras. 69. Item 66, the autonomous delivery vehicle, the multiple short-range sensors include an IR projector, two image sensors, and an RGB sensor. 70. Item 66, the autonomous delivery vehicle, the multiple short-range sensors are radar sensors. 71. Item 66, the autonomous delivery vehicle, the short-range sensors provide RGB-D data to the controller. 72. Item 66, the autonomous delivery vehicle, the controller determines the geometry of the road surface based on the RGB-D data received from the multiple short-range sensors. 73. Item 66, the autonomous delivery vehicle, the multiple short-range sensors detect objects within 4 meters of the autonomous delivery vehicle, and the long-range sensor suite detects objects more than 4 meters from the autonomous delivery vehicle.

74. An autonomous delivery vehicle comprising: a power base including at least two powered rear wheels, a caster front wheel, and an energy storage device, the power base configured to move at a commanded speed; a cargo platform including a plurality of short-range sensors, the cargo platform mechanically attached to the power base; a cargo container with a volume for receiving one or more objects to be delivered, the cargo container mounted on the cargo platform; a long-range sensor suite including a LIDAR and one or more cameras, the long-range sensor suite being mounted on the cargo container; and a controller for receiving data from the long-range sensor suite and the plurality of short-range sensors. 75. The autonomous delivery vehicle of claim 74, wherein the plurality of short-range sensors detect at least one characteristic of a drivable surface. 76. The autonomous delivery vehicle of claim 74, wherein the plurality of short-range sensors are stereo cameras. 77. 78. The autonomous delivery vehicle of claim 74, wherein the short-range sensors include an IR projector, two image sensors, and an RGB sensor. 79. The autonomous delivery vehicle of claim 74, wherein the short-range sensors are radar sensors. 79. The autonomous delivery vehicle of claim 74, wherein the short-range sensors provide RGB-D data to the controller. 80. The autonomous delivery vehicle of claim 74, wherein the controller determines a geometry of the road surface based on the RGB-D data received from the short-range sensors. 81. The autonomous delivery vehicle of claim 74, wherein the short-range sensors detect objects within 4 meters of the autonomous delivery vehicle and the long-range sensor suite detects objects greater than 4 meters from the autonomous delivery vehicle. 82. The autonomous delivery vehicle of claim 74, further comprising a second set of powered wheels that can engage the ground while the caster wheels are lifted off the ground.

83. An autonomous delivery vehicle comprising: a power base including at least two powered rear wheels, a caster front wheel, and an energy storage device, the power base configured to move at a commanded speed; a cargo platform, the cargo platform mechanically attached to the power base; and a short-range camera assembly mounted on the cargo platform that detects at least one characteristic of the drivable surface, the short-range camera assembly comprising a camera, a first light, and a first liquid-cooled heat sink, the first liquid-cooled heat sink cooling the first light and the camera. 84. The autonomous delivery vehicle of claim 83, wherein the short-range camera assembly further comprises a thermoelectric cooler between the camera and the liquid-cooled heat sink. 85. The autonomous delivery vehicle of claim 83, wherein the first light and the camera are recessed within a cover with an opening that deflects illumination from the first light away from the camera. 86. The autonomous delivery vehicle of claim 83, wherein the lights are angled downward at least 15° and recessed at least 4 mm into the cover to minimize distracting illumination to pedestrians. 87. The autonomous delivery vehicle of claim 83, wherein the camera has a field of view and the first light comprises two LEDs with a lens to generate two beams of light that are diffused to illuminate the field of view of the camera. 88. The autonomous delivery vehicle of claim 87, wherein the lights are angled approximately 50° apart and the lens generates a 60° beam. 89. The autonomous delivery vehicle of claim 83, wherein the short-range camera assembly includes an ultrasonic sensor mounted above the camera. 90. The autonomous delivery vehicle of claim 83, wherein the short-range camera assembly is mounted in a central position on the front of the cargo platform. 91. 92. The method of claim 22, wherein the historical data comprises surface data. 93. The method of claim 22, wherein the historical data comprises discontinuity data. 94. The autonomous delivery vehicle of claim 83, further comprising at least one corner camera assembly mounted on at least one corner of a front of the cargo platform, the at least one corner camera assembly comprising an ultrasonic sensor, a corner camera, a second light, and a second liquid cooled heat sink that cools the second light and the corner camera. 92. The method of claim 22, wherein the historical data comprises surface data. 93. The method of claim 22, wherein the historical data comprises discontinuity data.
The present invention provides, for example, the following:
(Item 1)
1. An autonomous delivery vehicle, comprising:
a power base including two powered front wheels, two powered rear wheels, and an energy storage device, the power base configured to move at a commanded speed and in a commanded direction to effect transportation of at least one object;
a cargo platform including a plurality of short range sensors, said cargo platform being mechanically attached to said power base;
a cargo container with a volume for receiving the at least one object, the cargo container being mounted on the cargo platform; and
a long range sensor suite comprising a LIDAR and one or more cameras, the long range sensor suite being mounted on the cargo container;
a controller for receiving data from the long range sensor suite and the plurality of short range sensors, the controller determining the commanded speed and the commanded direction based at least on the data, the controller providing the commanded speed and the commanded direction to the power base to complete the transport; and
An autonomous delivery vehicle comprising:
(Item 2)
2. The autonomous delivery vehicle of claim 1, wherein the data from the plurality of short-range sensors comprises at least one characteristic of a surface over which the power base travels.
(Item 3)
2. The autonomous delivery vehicle of claim 1, wherein the plurality of short-range sensors comprises at least one stereo camera.
(Item 4)
2. The autonomous delivery vehicle of claim 1, wherein the multiple short-range sensors comprise at least one IR projector, at least one image sensor, and at least one RGB sensor.
(Item 5)
2. The autonomous delivery vehicle of claim 1, wherein the plurality of short-range sensors comprises at least one radar sensor.
(Item 6)
2. The autonomous delivery vehicle of claim 1, wherein the data from the plurality of short-range sensors comprises RGB-D data.
(Item 7)
2. The autonomous delivery vehicle of claim 1, wherein the controller determines a geometry of a road surface based on RGB-D data received from the plurality of short-range sensors.
(Item 8)
2. The autonomous delivery vehicle of claim 1, wherein the plurality of short-range sensors detect objects within 4 meters of the AV and the long-range sensor suite detects objects greater than 4 meters from the autonomous delivery vehicle.
(Item 9)
2. The autonomous delivery vehicle of claim 1, wherein the plurality of short-range sensors comprises a cooling circuit.
(Item 10)
2. The autonomous delivery vehicle of claim 1, wherein the plurality of short-range sensors comprises ultrasonic sensors.
(Item 11)
The controller:
and an executable code, the executable code comprising:
accessing a map, the map being formed by a map processor, the map processor comprising:
a first processor that accesses point cloud data from the long range sensor suite, the point cloud data representing the surface;
a filter for filtering the point cloud data;
a second processor for forming a processable portion from the filtered point cloud data;
a third processor for merging the processable portions into at least one polygon;
a fourth processor for locating and labeling the at least one substantially discontinuous surface feature (SDSF) within the at least one polygon, if present, the locating and labeling forming labeled point cloud data;
a fifth processor for generating graphed polygons from the labeled point cloud data;
a sixth processor for determining a path from a start point to an end point based at least on the graphed polygon, the AV traversing the at least one SDSF along the path;
3. The autonomous delivery vehicle of claim 2, comprising:
(Item 12)
The filter comprises:
conditionally removing points representing transient objects and points representing outliers from the point cloud data;
replacing the removed points with a preselected height; and
12. The autonomous delivery vehicle of claim 11, further comprising a seventh processor that executes code including:
(Item 13)
The second processor comprises:
Segmenting the point cloud data into the processable portions;
removing points of a preselected height from said addressable portion;
12. The autonomous delivery vehicle of claim 11, comprising the executable code comprising:
(Item 14)
The third processor,
Reducing the size of the processable portion by analyzing outliers, voxels, and normals;
Enlarging an area from the reduced size addressable portion; and
determining an initial drivable surface from the expanded region; and
segmenting and meshing the initial drivable surface;
Locating polygons within the segmented and meshed initial drivable surface; and
establishing at least one drivable surface based at least on said polygon;
12. The autonomous delivery vehicle of claim 11, comprising the executable code comprising:
(Item 15)
The fourth processor
sorting the initial drivable surface point cloud data according to an SDSF filter, the SDSF filter including at least three categories of points;
locating at least one SDSF point based on whether the at least three categories of points, in combination, satisfy at least one first preselected criterion;
15. The autonomous delivery vehicle of claim 14, comprising the executable code comprising:
(Item 16)
16. The autonomous delivery vehicle of claim 15, wherein the fourth processor comprises the executable code to create at least one SDSF trajectory based on whether a plurality of the at least one SDSF point, in combination, satisfies at least one second preselected criterion.
(Item 17)
Creating a graphing polygon is
creating at least one polygon from the at least one drivable surface, the at least one polygon including an outer edge;
smoothing the outer edge; and
forming a running margin based on the smoothed outer edge;
adding said at least one SDSF track to said at least one drivable surface;
removing an inner edge from the at least one drivable surface according to at least one third preselected criterion; and
15. The autonomous delivery vehicle of claim 14, further comprising an eighth processor comprising the executable code comprising:
(Item 18)
20. The autonomous delivery vehicle of claim 17, further comprising a ninth processor comprising the executable code, wherein smoothing the outer edge comprises trimming the outer edge outward to form an outward edge.
(Item 19)
20. The autonomous delivery vehicle of claim 18, further comprising a tenth processor comprising the executable code, wherein forming the smoothed outer edge running margin comprises inwardly trimming the outward edge.
(Item 20)
The controller:
a subsystem for navigating at least one substantially discontinuous surface feature (SDSF) encountered by the autonomous delivery vehicle (AV), the AV navigating a path across a surface, the surface including the at least one SDSF, the path including a start point and an end point, the subsystem comprising:
a first processor that accesses a route configuration, the route configuration including at least one graphed polygon including filtered point cloud data, the filtered point cloud data including labeled features, and the point cloud data including a driveable margin;
a second processor for transforming the point cloud data into a global coordinate system;
a third processor for determining a boundary of the at least one SDSF, the third processor creating an SDSF buffer of a preselected size around the boundary;
a fourth processor for determining the at least one SDSF that may be traversed based on at least one SDSF traversal criterion; and
a fifth processor that creates an edge/weighted graph based on at least the at least one SDSF traversal criterion, the transformed point cloud data, and the root morphology;
a base controller that selects the path from the start point to the end point based at least on the edge/weight graph;
2. The autonomous delivery vehicle of claim 1, comprising:
(Item 21)
The at least one SDSF crossing criterion is
a preselected width of the at least one SDSF and a preselected smoothness of the at least one SDSF;
a minimum entry and exit distance between said at least one SDSF and said AV, said minimum entry and exit distance including a drivable surface;
the minimum approach distance between the at least one SDSF and the AV, accommodating an approximately 90° approach to the at least one SDSF by the AV;
21. The autonomous delivery vehicle of claim 20, comprising:
(Item 22)
1. A method for managing a global occupancy grid for an autonomous device, the global occupancy grid including global occupancy grid cells, the global occupancy grid cells associated with occupancy probabilities, the method comprising:
receiving sensor data from a sensor associated with the autonomous device;
creating a local occupancy grid based at least on the sensor data, the local occupancy grid having local occupancy grid cells;
When the autonomous device moves from a first area to a second area,
accessing historical data associated with the second area; and
creating a static grid based at least on the historical data;
moving the global occupancy grid to maintain the autonomous device in a central position of the global occupancy grid;
updating the moved global occupancy grid based on the static grid;
marking at least one of the global occupancy grid cells as unoccupied if the at least one of the global occupancy grid cells matches a location of the autonomous device;
For each said locally occupied grid cell,
calculating a position of the local occupancy grid cell on the global occupancy grid;
accessing a first occupancy probability from the global occupancy grid cells at the location;
accessing a second occupancy probability from the local occupancy grid cell at the location; and
calculating a new occupancy probability at the location on the global occupancy grid based on at least the first occupancy probability and the second occupancy probability;
A method comprising:
(Item 23)
23. The method of claim 22, further comprising range checking the new occupancy probabilities.
(Item 24)
The range checking comprises:
if the new occupancy probability is <0, setting the new occupancy probability to 0;
if the new occupancy probability is >1, setting the new occupancy probability to 1;
24. The method of claim 23, comprising:
(Item 25)
23. The method of claim 22, further comprising setting the global occupied grid cells to the new occupancy probability.
(Item 26)
24. The method of claim 23, further comprising setting the global occupancy grid cells to the range-checked new occupancy probability.
(Item 27)
1. A method for creating and managing an occupancy grid, comprising:
Transforming the sensor measurements into a reference frame associated with the device by a local occupancy grid creation node;
creating a time-stamped measurement occupancy grid;
publishing the time-stamped measured occupancy grid as a local occupancy grid;
Creating a plurality of local occupancy grids;
creating a static occupancy grid based on surface characteristics in a repository, the surface characteristics being associated with a location of the device;
moving a global occupancy grid associated with the location of the device to maintain the device and the local occupancy grid approximately aligned with respect to the global occupancy grid;
adding information from the static occupancy grid to the global occupancy grid;
marking an area in the global occupancy grid that is currently occupied by the device as unoccupied;
For at least one cell in each local occupancy grid,
determining a location of the at least one cell within the global occupancy grid;
accessing a first value at the location;
determining a second value at the location based on a relationship between the first value and a cell value for the at least one cell in the local occupancy grid;
comparing the second value against a preselected probability range;
if the probability value is within the preselected probability range, setting the global occupancy grid with the new value;
A method comprising:
(Item 28)
28. The method of claim 27, further comprising publishing the global occupancy grid.
(Item 29)
28. The method of claim 27, wherein the surface characteristics comprise a surface type and a surface discontinuity.
(Item 30)
28. The method of claim 27, wherein the relationship includes summing.
(Item 31)
1. A system for creating and managing an occupancy grid, comprising:
a plurality of local grid creating nodes that create at least one local occupancy grid, the at least one local occupancy grid being associated with a location of a device, the at least one local occupancy grid including at least one cell;
a global occupancy grid manager that accesses the at least one local occupancy grid;
Equipped with
The global occupancy grid manager:
creating a static occupancy grid based on surface characteristics in a repository, the surface characteristics being associated with a location of the device;
moving a global occupancy grid associated with the location of the device to maintain the device and the at least one local occupancy grid substantially aligned with the global occupancy grid;
adding information from the static occupancy grid to at least one global occupancy grid;
marking an area in the global occupancy grid that is currently occupied by the device as unoccupied;
For each of the at least one cell in each local occupancy grid,
determining a location of the at least one cell within the global occupancy grid;
accessing a first value at the location;
determining a second value at the location based on a relationship between the first value and a cell value for the at least one cell in the local occupancy grid;
comparing the second value against a preselected probability range;
if the probability value is within the preselected probability range, setting the global occupancy grid with the new value;
A system that performs the above.
(Item 32)
1. A method for updating a global occupancy grid, comprising:
When an autonomous device moves to a new location, updating the global occupancy grid with information from a static grid associated with the new location;
analyzing the surface at the new location; and
if the surface is drivable, updating the surface and updating the global occupancy grid with the updated surface;
updating the global occupancy grid with values from a repository of static values, the static values being associated with the new location;
A method comprising:
(Item 33)
Renewing the surface comprises:
accessing a local occupancy grid associated with the new location; and
For each cell in the local occupancy grid,
accessing a local occupancy grid surface classification confidence value and a local occupancy grid surface classification;
if the local occupancy grid surface classification is the same as the global surface classification in the global occupancy grid in the cell, adding the global surface classification confidence value in the global occupancy grid to the local occupancy grid surface classification confidence value to form a sum and using the sum to update the global occupancy grid in the cell;
if the local occupancy grid surface classification is not identical to the global surface classification in the global occupancy grid in the cell, subtracting the local occupancy grid surface classification confidence value from the global surface classification confidence value in the global occupancy grid to form a difference and updating the global occupancy grid with the difference;
if the difference is less than zero, updating the global occupancy grid with the local occupancy grid surface classification;
33. The method of claim 32, comprising:
(Item 34)
updating the global occupancy grid with the values from the repository of static values comprises:
For each cell in the local occupancy grid,
accessing a log-odds value, which is a local occupancy grid probability that the cell from the local occupancy grid is an occupancy value;
updating the log-odds value in the global occupancy grid with the local occupancy grid log-odds value in the cell;
decreasing the log-odds that the cell is occupied in the local occupancy grid if a preselected certainty that the cell is unoccupied is met and if the autonomous device is proceeding within a lane barrier and a local occupancy grid surface classification indicates a drivable surface;
increasing the log-odds within the local occupancy grid if the autonomous device expects to encounter a relatively uniform surface and if the local occupancy grid surface classification indicates a relatively non-uniform surface;
decreasing the log-odds within the local occupancy grid if the autonomous device expects to encounter a relatively uniform surface and if the local occupancy grid surface classification indicates a relatively uniform surface;
33. The method of claim 32, comprising:
(Item 35)
1. A method for real-time control of a configuration of a device, the device including a chassis, at least four wheels, a first side of the chassis operatively coupled to at least one of the at least four wheels, and an opposing second side of the chassis operatively coupled to at least one of the at least four wheels, the method comprising:
generating a map based on at least the prior surface features and an occupancy grid, the map being generated in non-real-time, the map including at least one location, the at least one location associated with at least one surface feature, the at least one surface feature associated with at least one surface classification and at least one mode;
determining a current surface characteristic as the device progresses;
updating the occupancy grid in real time with the current surface features; and
determining from the occupancy grid and the map a path that the device may take to traverse the at least one surface feature;
A method comprising:
(Item 36)
1. A method for real-time control of a configuration of a device, the device including a chassis, at least four wheels, a first side of the chassis operatively coupled to at least one of the at least four wheels, and an opposing second side of the chassis operatively coupled to at least one of the at least four wheels, the method comprising:
Receiving environmental data;
determining a surface type based at least on the environmental data;
determining a mode based on at least the surface type and a first configuration;
determining a second configuration based on at least the mode and the surface type;
determining a movement command based on at least the second configuration;
controlling a configuration of the device by using the move command to change the device from the first configuration to the second configuration;
A method comprising:
(Item 37)
37. The method of claim 36, wherein the environmental data comprises RGB-D image data.
(Item 38)
populating an occupancy grid based on at least the surface type and the mode;
determining the movement command based at least on the occupancy grid;
37. The method of claim 36, further comprising:
(Item 39)
40. The method of claim 38, wherein the occupancy grid comprises at least information based on data from at least one image sensor.
(Item 40)
37. The method of claim 36, wherein the environmental data comprises a road surface morphology.
(Item 41)
37. The method of claim 36, wherein the configuration comprises two clustered pairs of the at least four wheels, a first of the two pairs being positioned on the first side and a second of the two pairs being positioned on the second side, the first pair including a first front wheel and a first rear wheel, and the second pair including a second front wheel and a second rear wheel.
(Item 42)
42. The method of claim 41, wherein controlling the configuration includes coordinated powering of the first pair and the second pair based at least on the environmental data.
(Item 43)
42. The method of claim 41, wherein controlling the configuration includes transitioning from driving the at least four wheels and a pair of casters retracted, the pair of casters operably coupled to the chassis, to driving two wheels with the clustered first pair and the clustered second pair rotated to lift the first front wheel and the second front wheel, and the device is resting on the first rear wheel, the second rear wheel, and the pair of casters.
(Item 44)
42. The method of claim 41, wherein controlling the configuration includes rotating a pair of clusters operatively coupled to a first two powered wheels on the first side and a second two powered wheels on the second side based on at least the environmental data.
(Item 45)
37. The method of claim 36, wherein the device further comprises a cargo container, the cargo container mounted on the chassis, and the chassis controls a height of the cargo container.
(Item 46)
46. The method of claim 45, wherein the height of the cargo container is based at least on the environmental data.
(Item 47)
1. A system for real-time control of a configuration of a device, the device including a chassis, at least four wheels, a first side of the chassis, and an opposing second side of the chassis, the system comprising:
a device processor receiving real-time environmental data surrounding the device, the device processor determining a surface type based at least on the environmental data, the device processor determining a mode based at least on the surface type and a first configuration, and the device processor determining a second configuration based at least on the mode and the surface type;
a power base processor that determines a movement command based on at least the second configuration, the power base processor using the movement command to control a configuration of the device to change the device from the first configuration to the second configuration;
A system comprising:
(Item 48)
48. The system of claim 47, wherein the environmental data comprises RGB-D image data.
(Item 49)
48. The system of claim 47, wherein the device processor populates an occupancy grid based on at least the surface type and the mode.
(Item 50)
50. The system of claim 49, wherein the power base processor determines the movement commands based on at least the occupancy grid.
(Item 51)
50. The system of claim 49, wherein the occupancy grid comprises at least information based on data from at least one image sensor.
(Item 52)
Item 48. The system of item 47, wherein the environmental data comprises a road surface morphology.
(Item 53)
Item 48. The system of item 47, wherein the configuration comprises two clustered pairs of the at least four wheels, a first pair of the two pairs being positioned on the first side and a second pair of the two pairs being positioned on the second side, the first pair having a first front wheel and a first rear wheel, and the second pair having a second front wheel and a second rear wheel.
(Item 54)
54. The system of claim 53, wherein controlling the configuration includes coordinated powering of the first pair and the second pair based at least on the environmental data.
(Item 55)
54. The system of claim 53, wherein controlling the configuration includes transitioning from driving the at least four wheels and a pair of casters retracted, the pair of casters operably coupled to the chassis, to driving two wheels with the clustered first pair and the clustered second pair rotated to lift the first front wheel and the second front wheel, and the device is resting on the first rear wheel, the second rear wheel, and the pair of casters.
(Item 56)
1. A method for maintaining a global occupancy grid, comprising:
Locating a first position of the autonomous device;
When the autonomous device moves to a second location, the second location is associated with the global occupancy grid and the local occupancy grid;
updating the global occupancy grid with at least one occupancy probability value associated with the first location;
updating the global occupancy grid with at least one drivable surface associated with the local occupancy grid;
updating the global occupancy grid with a surface confidence associated with the at least one drivable surface;
updating the global occupancy grid with log-odds of the at least one occupancy probability value using a first Bayes function;
adjusting the log-odds based on a characteristic associated with at least the second location; and
when the autonomous device remains in the first location and the global occupancy grid and the local occupancy grid are co-located;
updating the global occupancy grid with the at least one drivable surface associated with the local occupancy grid;
updating the global occupancy grid with the surface confidence associated with the at least one drivable surface;
updating the global occupancy grid with log-odds of the at least one occupancy probability value using a second Bayes function;
adjusting the log odds based on a characteristic associated with at least the second location; and
A method comprising:
(Item 57)
Creating the map includes:
accessing point cloud data representing the surface;
filtering the point cloud data; and
forming the filtered point cloud data into a processable portion; and
merging the processable portions into at least one concave polygon;
locating and labeling the at least one SDSF within the at least one concave polygon, the locating and labeling forming labeled point cloud data;
generating a graphing polygon based at least on the at least one concave polygon;
determining the path from a start point to an end point based at least on the graphed polygon, the AV traversing the at least one SDSF along the path;
36. The method of claim 35, comprising:
(Item 58)
Filtering the point cloud data includes:
conditionally removing points representing transient objects and points representing outliers from the point cloud data;
replacing the removed points with a preselected height; and
58. The method of claim 57, comprising:
(Item 59)
Forming the processing portion includes
Segmenting the point cloud data into the processable portions;
removing points of a preselected height from said addressable portion;
58. The method of claim 57, comprising:
(Item 60)
Merging the processable portions comprises:
Reducing the size of the processable portion by analyzing outliers, voxels, and normals;
Enlarging an area from the reduced size addressable portion; and
determining an initial drivable surface from the expanded region; and
segmenting and meshing the initial drivable surface;
Locating polygons within the segmented and meshed initial drivable surface; and
establishing at least one drivable surface based at least on said polygon;
58. The method of claim 57, comprising:
(Item 61)
Locating and labeling the at least one SDSF comprises:
sorting the initial drivable surface point cloud data according to an SDSF filter, the SDSF filter including at least three categories of points;
locating at least one SDSF point based on whether the at least three categories of points, in combination, satisfy at least one first preselected criterion;
61. The method of claim 60, comprising:
(Item 62)
62. The method of claim 61, further comprising generating at least one SDSF trajectory based at least on whether a plurality of the at least one SDSF point, in combination, satisfies at least one second preselected criterion.
(Item 63)
Creating the graphing polygon further comprises:
creating at least one polygon from the at least one drivable surface, the at least one polygon including an outer edge;
smoothing the outer edge; and
forming a running margin based on the smoothed outer edge;
adding said at least one SDSF track to said at least one drivable surface;
removing an inner edge from the at least one drivable surface according to at least one third preselected criterion; and
Item 63. The method of item 62, comprising:
(Item 64)
Item 64. The method of item 63, wherein smoothing the outer edge comprises trimming the outer edge outwardly to form an outward edge.
(Item 65)
64. The method of claim 63, wherein forming the smoothed outer edge running margin comprises inwardly trimming the outwardly facing edge.
(Item 66)
1. An autonomous delivery vehicle, comprising:
a power base including two powered front wheels, two powered rear wheels, and an energy storage device, said power base configured to move at a commanded speed;
a cargo platform including a plurality of short range sensors, said cargo platform being mechanically attached to said power base;
a cargo container with a volume for receiving one or more objects to be delivered, said cargo container being mounted on said cargo platform;
a long range sensor suite comprising a LIDAR and one or more cameras, the long range sensor suite being mounted on the cargo container;
a controller for receiving data from the long range sensor suite and the plurality of short range sensors;
An autonomous delivery vehicle comprising:
(Item 67)
67. The autonomous delivery vehicle of claim 66, wherein the plurality of short-range sensors detect at least one characteristic of a drivable surface.
(Item 68)
67. The autonomous delivery vehicle of claim 66, wherein the multiple short-range sensors are stereo cameras.
(Item 69)
Item 67. The autonomous delivery vehicle of item 66, wherein the multiple short-range sensors comprise an IR projector, two image sensors, and an RGB sensor.
(Item 70)
Item 67. The autonomous delivery vehicle of item 66, wherein the plurality of short-range sensors are radar sensors.
(Item 71)
Item 67. The autonomous delivery vehicle of item 66, wherein the short-range sensor provides RGB-D data to the controller.
(Item 72)
Item 67. The autonomous delivery vehicle of item 66, wherein the controller determines a geometry of a road surface based on RGB-D data received from the multiple short-range sensors.
(Item 73)
67. The autonomous delivery vehicle of claim 66, wherein the plurality of short-range sensors detect objects within 4 meters of the autonomous delivery vehicle and the long-range sensor suite detects objects greater than 4 meters from the autonomous delivery vehicle.
(Item 74)
1. An autonomous delivery vehicle, comprising:
a power base including at least two powered rear wheels, a caster front wheel, and an energy storage device, the power base configured to move at a commanded speed;
a cargo platform including a plurality of short range sensors, said cargo platform being mechanically attached to said power base;
a cargo container with a volume for receiving one or more objects to be delivered, said cargo container being mounted on said cargo platform;
a long range sensor suite comprising a LIDAR and one or more cameras, the long range sensor suite being mounted on the cargo container;
a controller for receiving data from the long range sensor suite and the plurality of short range sensors;
An autonomous delivery vehicle comprising:
(Item 75)
75. The autonomous delivery vehicle of claim 74, wherein the plurality of short-range sensors detect at least one characteristic of a drivable surface.
(Item 76)
75. The autonomous delivery vehicle of claim 74, wherein the multiple short-range sensors are stereo cameras.
(Item 77)
75. The autonomous delivery vehicle of claim 74, wherein the multiple short-range sensors comprise an IR projector, two image sensors, and an RGB sensor.
(Item 78)
75. The autonomous delivery vehicle of claim 74, wherein the plurality of short-range sensors are radar sensors.
(Item 79)
Item 75. The autonomous delivery vehicle of item 74, wherein the short-range sensor provides RGB-D data to the controller.
(Item 80)
Item 75. The autonomous delivery vehicle of item 74, wherein the controller determines a geometry of a road surface based on RGB-D data received from the plurality of short-range sensors.
(Item 81)
75. The autonomous delivery vehicle of claim 74, wherein the plurality of short-range sensors detect objects within 4 meters of the autonomous delivery vehicle and the long-range sensor suite detects objects greater than 4 meters from the autonomous delivery vehicle.
(Item 82)
75. The autonomous delivery vehicle of claim 74, further comprising a second set of powered wheels that can engage the ground while the caster wheels are lifted off the ground.
(Item 83)
1. An autonomous delivery vehicle, comprising:
a power base including at least two powered rear wheels, a caster front wheel, and an energy storage device, the power base configured to move at a commanded speed;
a cargo platform, the cargo platform being mechanically attached to the power base;
a short-range camera assembly mounted to the cargo platform for detecting at least one characteristic of a drivable surface;
Equipped with
The short range camera assembly includes:
A camera and
The first light,
a first liquid-cooled heat sink;
Equipped with
The first liquid cooled heat sink cools the first light and the camera.
(Item 84)
84. The autonomous delivery vehicle of claim 83, wherein the short-range camera assembly further comprises a thermoelectric cooler between the camera and the liquid-cooled heat sink.
(Item 85)
Item 84. The autonomous delivery vehicle of item 83, wherein the first light and the camera are embedded in a cover with an opening that deflects illumination from the first light away from the camera.
(Item 86)
Item 84. The autonomous delivery vehicle of item 83, wherein the lights are angled at least 15 degrees downward and recessed at least 4 mm into the cover to minimize distracting illumination to pedestrians.
(Item 87)
Item 84. The autonomous delivery vehicle of item 83, wherein the camera has a field of view and the first light comprises two LEDs with lenses for generating two beams of light that are diffused to illuminate the field of view of the camera.
(Item 88)
Item 88. The autonomous delivery vehicle of item 87, wherein the lights are angled approximately 50 degrees apart and the lenses produce a 60 degree beam.
(Item 89)
Item 84. The autonomous delivery vehicle of item 83, wherein the short-range camera assembly includes an ultrasonic sensor mounted above the camera.
(Item 90)
Item 84. The autonomous delivery vehicle of item 83, wherein the short-range camera assembly is mounted in a central position on the front of the cargo platform.
(Item 91)
and at least one corner camera assembly mounted on at least one corner of a front surface of the cargo platform, the at least one corner camera assembly comprising:
An ultrasonic sensor;
Corner camera and
A second light,
a second liquid-cooled heat sink, the second liquid-cooled heat sink cooling the second light and the corner camera;
Item 84. The autonomous delivery vehicle of item 83, comprising:
(Item 92)
23. The method of claim 22, wherein the historical data comprises surface data.
(Item 93)
23. The method of claim 22, wherein the historical data comprises discontinuity data.

The present teachings will be more readily understood with reference to the following description taken in conjunction with the accompanying drawings.

FIG. 1-1 is a schematic block diagram of the major components of the system of the present teachings.

1-2 is a schematic block diagram of the major components of the map processor of the present teachings.

1-3 are schematic block diagrams of the major components of the perception processor of the present teachings.

1-4 are schematic block diagrams of the major components of an autonomy processor of the present teachings.

FIG. 1A is a schematic block diagram of a system of the present teachings for preparing a travel path for an AV.

FIG. 1B is a pictorial diagram of an exemplary configuration of a device incorporating a system of the present teachings.

FIG. 1C is a side view of an automated delivery vehicle showing the fields of view of several long-range and short-range sensors.

FIG. 1D is a schematic block diagram of a map processor of the present teachings.

FIG. 1E is a pictorial diagram of a first portion of the map processor flow of the present teachings.

FIG. 1F is an image of a segmented point cloud of the present teachings.

FIG. 1G is a pictorial diagram of a second portion of the map processor of the present teachings.

FIG. 1H is an image of a drivable surface detection result of the present teachings.

FIG. 1I is a pictorial diagram of the SDSF detector flow of the present teachings.

FIG. 1J is a pictorial representation of the SDSF category of the present teachings.

FIG. 1K is an image of an SDSF identified by the system of the present teachings.

1L and 1M are pictorial diagrams of the polygon processing of the present teachings. 1L and 1M are pictorial diagrams of the polygon processing of the present teachings.

FIG. 1N is an image of the polygons and SDSF identified by the system of the present teachings.

FIG. 2A is an isometric view of an autonomous vehicle of the present teachings.

FIG. 2B is a top view of a cargo container showing the field of view of selected long-range sensors.

2C-2F are diagrams of a long range sensor assembly. 2C-2F are diagrams of a long range sensor assembly. 2C-2F are diagrams of a long range sensor assembly. 2C-2F are diagrams of a long range sensor assembly.

FIG. 2G is a top view of a cargo container showing the field of view of selected short-range sensors.

FIG. 2H is an isometric view of a cargo platform of the present teachings.

2I-2L are isometric views of the short range sensor. 2I-2L are isometric views of the short range sensor. 2I-2L are isometric views of the short range sensor. 2I-2L are isometric views of the short range sensor.

2M-2N are isometric views of an autonomous vehicle of the present teachings. 2M-2N are isometric views of an autonomous vehicle of the present teachings.

2O-2P are isometric views of an autonomous vehicle of the present teachings with the exterior panels removed. 2O-2P are isometric views of an autonomous vehicle of the present teachings with the exterior panels removed.

FIG. 2Q is an isometric view of an autonomous vehicle of the present teachings with a portion of the top panel removed.

2R-2V are diagrams of long range sensors on an autonomous vehicle of the present teachings. 2R-2V are diagrams of long range sensors on an autonomous vehicle of the present teachings. 2R-2V are diagrams of long range sensors on an autonomous vehicle of the present teachings. 2R-2V are diagrams of long range sensors on an autonomous vehicle of the present teachings. 2R-2V are diagrams of long range sensors on an autonomous vehicle of the present teachings.

2W-2Z are diagrams of an ultrasonic sensor. 2W-2Z are diagrams of an ultrasonic sensor. 2W-2Z are diagrams of an ultrasonic sensor. 2W-2Z are diagrams of an ultrasonic sensor.

2AA-2BB are diagrams of the central short-range camera assembly. 2AA-2BB are diagrams of the central short-range camera assembly .

FIG. 3A is a schematic block diagram of a system in one configuration of the present teachings.

FIG. 3B is a schematic block diagram of a system in another configuration of the present teachings.

FIG. 3C is a schematic block diagram of a system of the present teachings that can initially create a global occupancy grid.

FIG. 3D is a pictorial representation of a static grid of the present teachings.

3E and 3F are pictorial representations of the creation of an occupancy grid of the present teachings. 3E and 3F are pictorial representations of the creation of an occupancy grid of the present teachings.

FIG. 3G is a pictorial representation of a priori occupancy grid of the present teachings.

FIG. 3H is a pictorial representation of updating the global occupancy grid of the present teachings.

FIG. 3I is a flow diagram of a method of the present teachings for publishing a global occupancy grid.

FIG. 3J is a flow diagram of a method of the present teachings for updating the global occupancy grid.

3K-3M are a flow diagram of another method of the present teachings for updating the global occupancy grid. 3K-3M are a flow diagram of another method of the present teachings for updating the global occupancy grid. 3K-3M are a flow diagram of another method of the present teachings for updating the global occupancy grid.

4A-4C are perspective pictorial views of devices of the present teachings mounted in various modes.

FIG. 4B is a schematic block diagram of a system of the present teachings.

FIG. 4C is a schematic block diagram of the driving surface processor component of the present teachings.

FIG. 4D is a schematic block/pictorial flow diagram of the process of the present teachings.

4E and 4F are perspective and side views, respectively, of a configuration of a device of the present teachings in a standard mode. 4E and 4F are perspective and side views, respectively, of a configuration of a device of the present teachings in a standard mode.

4G and H are perspective and side views, respectively, of a configuration of a device of the present teachings in four-wheel mode. 4G and H are perspective and side views, respectively, of a configuration of a device of the present teachings in four-wheel mode.

4I and 4J are perspective and side views, respectively, of a configuration of a device of the present teachings in an elevated four-wheel mode. 4I and 4J are perspective and side views, respectively, of a configuration of a device of the present teachings in an elevated four-wheel mode.

FIG. 4K is a flowchart of the method of the present teachings.

FIG. 5A is a schematic block diagram of a device controller of the present teachings.

FIG. 5B is a schematic block diagram of an SDSF processor of the present teachings.

FIG. 5C is an image of the SDSF approach identified by the system of the present teachings.

FIG. 5D is an image of the route configuration produced by the system of the present teachings.

FIG. 5E is a schematic block diagram of a mode of the present teachings.

5F-5J are flow charts of methods of the present teachings for traversing an SDSF. 5F-5J are flow charts of methods of the present teachings for traversing an SDSF. 5F-5J are flow charts of methods of the present teachings for traversing an SDSF. 5F-5J are flow charts of methods of the present teachings for traversing an SDSF. 5F-5J are flow charts of methods of the present teachings for traversing an SDSF.

FIG. 5K is a schematic block diagram of a system of the present teachings for traversing an SDSF.

5L-5N are pictorial representations of the method of FIGS. 5F-5H. 5L-5N are pictorial representations of the method of FIGS. 5F-5H. 5L-5N are pictorial representations of the method of FIGS. 5F-5H.

FIG. 5O is a pictorial representation of converting an image into a polygon.

Detailed Description
The systems and methods of the present teachings can use on-board sensors and previously developed maps to develop an occupancy grid and use these aids to navigate the AV across surface features, including reconfiguring the AV based on surface type and previous.

Now referring to FIG. 1-1, the AV system 100 may include a structure on which the sensors 10701 may be mounted and within which the device controller 10111 may be executed. The structure may include a power base 10112 that may direct the movement of wheels that are part of the structure and enable the movement of the AV. The device controller 10111 may be executed on at least one processor located on the AV and may receive data from the sensors 10701, which may be located on the AV, including but not limited to. The device controller 10111 may provide speed, direction, and configuration information to the base controller 10114, which may provide movement commands to the power base 10112. The device controller 10111 may receive map information from the map processor 10104, which may prepare a map of the area surrounding the AV. The device controller 10111 may include a sensor processor 10703 that may receive and process inputs from the sensors 10701, including but not limited to on-AV sensors. In some configurations, the device controller 10111 may include a perception processor 2143, an autonomy processor 2145, and a driver processor 2127. The perception processor 2143 may, for example, but not limited to, locate static and dynamic obstacles, determine traffic signal conditions, create occupancy grids, and classify surfaces. The autonomy processor 2145 may, for example, but not limited to, determine the maximum speed of the AV and determine the type of situation in which the AV is navigating, for example, on a road, on a sidewalk, at an intersection, and/or under remote control. The driver processor 2127 may, for example, but not limited to, create commands under the direction of the autonomy processor 2145 and transmit them on the base controller 10114.

1-2, the map processor 10104 can create a map of surface features and provide the map through the device controller 10111 to the perception processor 2143, which can update the occupancy grid. The map processor 10104 can include a feature extractor 10801, a point cloud organizer 10803, a transient processor 10805, a segmenter 10807, a polygon generator 10809, an SDSF line generator 10811, and a combiner 10813, among many other aspects. The feature extractor 10801 can include a first processor that accesses point cloud data representing the surface. The point cloud organizer 10803 can include a second processor that forms processable portions from the filtered point cloud data. The transient processor 10805 can include a first filter that filters the point cloud data. The segmenter 10807 may include executable code that may include, but is not limited to, segmenting the point cloud data into processable portions and removing points of a preselected height from the processable portions. The first filter may optionally include executable code that may include, but is not limited to, conditionally removing points representing transient objects and points representing outliers from the point cloud data and replacing the removed points with a preselected height. The polygon generator 10809 may include a third processor that merges the processable portions into at least one concave polygon. The third processor may optionally include executable code that may include, but is not limited to, reducing a size of the processable portion by analyzing outliers, voxels, and normals, growing a region from the reduced-sized processable portion, determining an initial drivable surface from the grown region, segmenting and meshing the initial drivable surface, locating polygons within the segmented and meshed initial drivable surface, and setting the drivable surface based at least on the polygons. The SDSF line generator 10811 may include a fourth processor that locates and labels at least one SDSF within at least one concave polygon, where the locating and labeling may form a labeled point cloud data. The fourth processor may optionally include executable code that may include, but is not limited to, sorting the drivable surface point cloud data according to an SDSF filter, where the SDSF filter includes at least three categories of points, and locating at least one SDSF point based at least on whether the categories of points in combination satisfy at least one first preselected criterion. The combiner 10813 may include a fifth processor that creates a graphing polygon. Creating a graphed polygon may optionally include executable code that may include, but is not limited to, creating at least one polygon from at least one drivable surface, the at least one polygon including an edge, smoothing the edge, forming a driving margin based on the smoothed edge, adding at least one SDSF trajectory to the at least one drivable surface, and removing the edge from the at least one drivable surface according to at least one third preselected criterion. Smoothing the edge may optionally include executable code that may include, but is not limited to, trimming the edge outward. Forming a driving margin of the smoothed edge may optionally include executable code that may include, but is not limited to, trimming the outward edge inward.

Now referring to FIG. 1-3, a map can be provided to the AV, which can include on-board sensors, powered wheels, and a processor for receiving sensor and map data and using the data to power configure the AV to traverse various types of surfaces, among other things, as the AV delivers goods, for example. The on-board sensors can capture an occupancy grid and can provide data that can be used to detect dynamic obstacles. The occupancy grid can also be captured by the map. The device controller 10111 can include a perception processor 2143 that can receive and process the sensor and map data and can use the data to update the occupancy grid.

Referring now to FIG. 1-4, the device controller 10111 can include a configuration processor 41023 that can automatically determine a configuration for the AV based on at least the mode of the AV and the surface characteristics encountered. The autonomy processor 2145 can include a control processor 40325 that can determine the type of surface that needs to be traversed and the configuration the AV needs to take to traverse the surface based on at least the map (planned route to be followed), information from the configuration processor 41023, and the mode of the AV. The autonomy processor 2145 can provide commands to the motor drive processor 40326 to implement the commands.

Now referring to FIG. 1A, the map processor 10104 can enable a device, such as, but not limited to, an AV or semi-autonomous device, to navigate within an environment that may include features such as SDSF. The features in the map, along with on-board sensors, can enable the AV to navigate over various surfaces. In particular, the SDSF can be precisely identified and labeled such that the AV can automatically maintain AV performance during entry and exit of the SDSF, and AV speed, configuration, and direction can be controlled for safe SDSF traversal.

Continuing with reference to FIG. 1A, in some configurations, the system 100 for managing the traversal of an SDSF can include AVs 10101, a core cloud infrastructure 10103, AV services 10105, a device controller 10111, sensors 10701, and a power base 10112. The AVs 10101 can provide transportation and shuttle services from origins to destinations, for example, but not limited to, following dynamically determined routes as modified by incoming sensor information. The AVs 10101 can include, but are not limited to, devices that have autonomous modes, devices that can operate fully autonomously, devices that can be operated at least partially remotely, and devices that can include combinations of those features. The transport device services 10105 can provide drivable surface information, including features, to the device controller 10111. The device controller 10111 can modify the drivable surface information according to at least, for example, but not limited to, incoming sensor information and feature traversal requirements, and can plan a route for the AV 10101 based on the modified drivable surface information. The device controller 10111 can submit commands to the power base 10112, which instructs the power base 10112 to provide speed, direction, and configuration commands to the wheel motors and cluster motors, which can cause the AV 10101 to follow the planned route and lift and lower its cargo accordingly. The transport device services 10105 can access route-related information from the core cloud infrastructure 10103, which can include, but is not limited to, storage and content distribution facilities. In some configurations, the core cloud infrastructure 10103 may include commercial products such as, for example, but not limited to, AMAZON WEB SERVICES®, GOOGLE CLOUD ^™ , and ORACLE CLOUD®.

1B, an exemplary AV that may include a device controller 10111 (FIG. 1A) that may receive information from the map processor 10104 (FIG. 1A) of the present teachings may include a power base assembly, such as, for example, but not limited to, a power base, fully described in U.S. Patent Application No. 16/035,205, filed July 13, 2018 and entitled "Mobility Device" or U.S. Patent No. 6,571,892, filed August 15, 2001 and entitled "Control System and Method," both of which are incorporated herein by reference in their entirety. Exemplary power base assemblies are described herein not to limit the present teachings, but instead to clarify features of any power base assembly that may be useful in implementing the techniques of the present teachings. The exemplary power base assembly can optionally include a power base 10112, a wheel cluster assembly 11100, and a payload carrier height assembly 10068. The exemplary power base assembly can optionally provide electrical and mechanical power to drive the wheels 11203 and the cluster 11100 that can raise and lower the wheels 11203. The power base 10112 can control the rotation of the cluster assembly 11100 and the elevation and lowering of the payload carrier height assembly 10068 to assist in the substantially discontinuous surface traversal of the present teachings. Other such devices can also be used to accommodate the SDSF detection and traversal of the present teachings.

1A again, in some configurations, sensors inside the exemplary power base can detect the orientation and rate of change of orientation of the AV10101, motors can enable servo operation, and a controller can interpret the information from the internal sensors and motors. Appropriate motor commands can be calculated to achieve vehicle performance and implement path following commands. Left and right wheel motors can drive wheels on both sides of the AV10101. In some configurations, the front and rear wheels can be coupled to drive together, such that the two left wheels can drive together and the two right wheels can drive together. In some configurations, turning can be accomplished by driving the left and right motors at different rates, and the cluster motors can rotate the wheel base in the forward/rearward direction. This can allow the AV10101 to remain level while the front wheels are higher or lower than the rear wheels. This feature can be useful, for example, but not limited to, when ascending or descending an SDSF. The payload carrier 10173 can be automatically raised and lowered based at least on the underlying terrain.

Continuing with reference to FIG. 1A, in some configurations, the point cloud data may include route information regarding the area through which the AV10101 should travel. Possibly, the point cloud data collected by a mapping device similar to or the same as the AV10101 may be time-tagged. The path along which the mapping device travels may be referred to as a mapped trajectory. The point cloud data processing described herein may be performed as the mapping device traverses the mapped trajectory or after point cloud data collection is completed. After the point cloud data are collected, they may undergo point cloud data processing, which may include initial filtering and point reduction, point cloud segmentation, and feature detection as described herein. In some configurations, the core cloud infrastructure 10103 may provide long-term or short-term storage for the collected point cloud data and may provide the data to the AV service 10105. The AV service 10105 can select among the possible point cloud datasets to find a dataset that covers an area surrounding a desired starting point for the AV 10101 and a desired destination for the AV 10101. The AV service 10105 can include a map processor 10104 that can reduce the size of the point cloud data and determine features represented in the point cloud data, without limitation. In some configurations, the map processor 10104 can determine the location of the SDSF from the point cloud data. In some configurations, polygons can be created from the point cloud data as a technique to segment the point cloud data and ultimately establish a drivable surface. In some configurations, the SDSF detection and drivable surface determination can proceed in parallel. In some configurations, the SDSF detection and drivable surface determination can proceed sequentially.

1C, in some configurations, the AV may be configured to deliver cargo and/or perform other functions that involve autonomously navigating to a desired location. In some applications, the AV may be remotely guided. In some configurations, the AV 20100 includes a cargo container that may be opened automatically, in response to user input, remotely, or manually to allow a user to place or remove luggage and other items. The cargo container 20110 is mounted on a cargo platform 20160 that is mechanically connected to a power base 20170. The power base 20170 includes four powered wheels 20174 and two caster wheels 20176. The power base provides speed and directional control to move the cargo container 20110 along the ground and over obstacles, including curbs and other discontinuous surface features.

With continued reference to FIG. 1C, the cargo platform 20160 is connected to the power base 20170 through two U-frames 20162. Each U-frame 20162 is rigidly attached to the structure of the cargo platform 20160 and includes two holes that allow a rotatable joint 20164 to be formed with the end of each arm 20172 on the power base 20170. The power base controls the rotational position of the arms and therefore the height and tilt angle of the cargo container 20110.

With continued reference to FIG. 1C, in some configurations, the AV20100 includes one or more processors for receiving data, navigating routes, and selecting the direction and speed of the power base 20170.

Now referring to FIG. 1D, in some configurations, the map processor 10104 of the present teachings can locate the SDSF on a map. The map processor 10104 can include, but is not limited to, a feature extractor 10801, a point cloud organizer 10803, a transient processor 10805, a segmenter 10807, a polygon generator 10809, an SDSF line generator 10811, and a data combiner 10813.

Continuing with reference to FIG. 1D, the feature extractor 10801 (FIG. 1-2) can include, but is not limited to, line-of-sight filtering 10121 of the point cloud data 10131 and the mapped trajectory 10133. Line-of-sight filtering can remove points that are hidden from the direct line of sight of the sensor that collects the point cloud data and forms the mapped trajectory. The point cloud organizer 10803 (FIG. 1-2) can organize (10151) the reduced point cloud data 10132, possibly according to preselected criteria associated with the specific features. In some configurations, the transient processor 10805 (FIG. 1-2) can remove (10153) transient points from the organized point cloud data and the mapped trajectory 10133 by any number of methods, including those described herein. Transient points can complicate processing, especially when the specific features are stationary. A segmenter 10807 (FIG. 1-2) can divide the processed point cloud data 10135 into processable chunks. In some configurations, the processed point cloud data 10135 can be segmented (10155) into sections having a preselected minimum number of points, for example, but not limited to, about 100,000 points. In some configurations, further point reduction can be based on preselected criteria that can be related to the features to be extracted. For example, if points above a certain height are not important to locating the features, those points can be deleted from the point cloud data. In some configurations, the height of at least one of the sensors collecting the point cloud data can be considered as an origin, and points above the origin can be removed from the point cloud data, for example, when only points of interest are associated with surface features. After the filtered point cloud data 10135 is segmented, forming segments 10137, the remaining points can be divided into drivable surface sections and surface features can be located. In some configurations, a polygon generator 10809 (FIG. 1-2) can locate drivable surfaces by generating (10161) polygons 10139, for example, but not limited to, as described herein. In some configurations, a SDSF line generator 10811 (FIG. 1-2) can locate surface features by generating (10163) SDSF lines 10141, for example, but not limited to, as described herein. In some configurations, a combiner 10813 (FIG. 1-2) can combine (10165) polygons 10139 and SDSF 10141 to create a data set that can be further processed to generate an actual path along which the AV 10101 (FIG. 1A) can proceed.

1E, removing 10153 (FIG. 1D) objects that are transient to the mapped trajectory 10133, such as the exemplary time-stamped point 10751, from the point cloud data 10131 (FIG. 1D) can include casting a ray 10753 from the time-stamped point on the mapped trajectory 10133 to each time-stamped point in the point cloud data 10131 (FIG. 1D) that has a substantially identical time stamp. If the ray 10753 intersects a point between the time-stamped point on the mapped trajectory 10133 and the end point of the ray 10753, e.g., point D10755, then the intersection point D10755 can be assumed to have entered the point cloud data during a different sweep of the camera. The intersection point, e.g., intersection point D10755, can be assumed to be part of a transient object and can be removed from the reduced point cloud data 10132 (FIG. 1D) as not representing a fixed feature, such as an SDSF. The result is, for example, but not limited to, processed point cloud data 10135 (FIG. 1D) that is free of transient objects. Points that were removed as part of a transient object, but that are also substantially at ground level, can be returned (10754) to the processed point cloud data 10135 (FIG. 1D). Transient objects may not include certain features, such as, for example, but not limited to, SDSF 10141 (FIG. 1D), and therefore can be removed without interfering with the integrity of the point cloud data 10131 (FIG. 1D) when SDSF 10141 (FIG. 1D) is a feature that is being detected.

Continuing with reference to FIG. 1E, segmenting 10155 (FIG. 1D) the processed point cloud data 10135 (FIG. 1D) can generate sections 10757 having a preselected size and shape, such as, but not limited to, a rectangle 10154 (FIG. 1F) having a minimum preselected side length and containing approximately 100,000 points. From each section 10757, points that are not necessarily relevant to the specific task, such as, but not limited to, points located above a preselected level, can be removed (10157) (FIG. 1D) to reduce the dataset size. In some configurations, the preselected level can be the height of AV10101 (FIG. 1A). Removing these points can lead to more efficient processing of the dataset.

Referring again primarily to FIG. 1D, the map processor 10104 can provide the device controller 10111 with at least one dataset that can be used to generate direction, speed, and configuration commands for controlling the AV 10101 (FIG. 1A). The at least one dataset can include points that can be connected to other points in the dataset, and each line connecting points in the dataset traverses the drivable surface. To determine such route points, the segmented point cloud data 10137 can be divided into polygons 10139, and the vertices of the polygons 10139 can potentially become route points. The polygons 10139 can include features such as, for example, SDSF 10141.

Continuing with reference to FIG. 1D, in some configurations, creating the processed point cloud data 10135 can include filtering the voxels. To reduce the number of points that will undergo future processing, in some configurations, the centroid of each voxel in the dataset can be used to approximate the points in the voxel, and all points except the centroid can be excluded from the point cloud data. In some configurations, the center of the voxel can be used to approximate the points in the voxel. Other methods for reducing the size of the filtered segment 10251 (FIG. 1G) can also be used, such as taking a random point subsample, such that a fixed number of points, selected uniformly at random, can be excluded from the filtered segment 10251 (FIG. 1G), for example and without limitation.

Continuing with still further reference to FIG. 1D, in some configurations, creating the processed point cloud data 10135 can include calculating normals from a dataset from which outliers have been removed and which has been reduced in size through voxel filtering. The normals for each point in the filtered dataset can be used for various processing possibilities, including curve reconstruction algorithms. In some configurations, estimating and filtering normals in the dataset can include obtaining an underlying surface from the dataset using a surface meshing technique and calculating normals from the surface mesh. In some configurations, estimating normals can include using approximations to infer surface normals directly from the dataset, such as, for example, but not limited to, determining normals to a fitting plane obtained by applying a total least squares method to k nearest neighbors of the point. In some configurations, the value of k can be chosen based at least on empirical data. Filtering normals can include removing any normals that are more than about 45° from perpendicular to the x-y plane. In some configurations, a filter can be used to align normals in the same direction. If a portion of the data set represents a planar surface, redundant information contained in adjacent normals can be filtered out, either by performing random subsampling or by filtering out a point from the set of associated points. In some configurations, selecting a point can include recursively decomposing the data set into boxes until each box contains at most k points. A single normal can be calculated from the k points in each box.

Continuing with reference to FIG. 1D, in some configurations, creating the processed point cloud data 10135 can include growing regions in the dataset by clustering points that geometrically fit with a surface representing the dataset, and refining the surface as the region grows to obtain the best approximation of the maximum number of points. Region growing can merge points in terms of a smoothness constraint. In some configurations, the smoothness constraint can be determined, for example, empirically or based on a desired surface smoothness. In some configurations, the smoothness constraint can include a range of about 10π/180 to about 20π/180. The output of region growing is a set of point clusters, each point cluster being a set of points, each of which is considered to be part of the same smooth surface. In some configurations, region growing can be based on a comparison of angles between normals. Region growing can be performed using a method such as, for example, but not limited to, region growing segmentation http://pointclouds. This can be accomplished by algorithms such as http://pointclouds.org/documentation/tutorials/region_growing_segmentation.php and http://pointclouds.org/documentation/tutorials/cluster_extraction.php#cluster-extraction.

1G, the segmented point cloud data 10137 (FIG. 1D) can be used to generate a polygon 10759, e.g., a 5m×5m polygon (10161) (FIG. 1D). The point sub-clusters can be converted to polygon 10759, e.g., using meshing. Meshing can be accomplished by standard methods such as, for example, but not limited to, marching cubes, marching tetrahedrons, surface nets, greedy meshing, and dual contouring. In some configurations, the polygon 10759 can be generated by projecting a local neighborhood of points along the normals of the points and connecting unconnected points. The resulting polygon 10759 can be based on at least the size of the neighborhood, the maximum allowable distance for the points to be considered, the maximum edge length for the polygon, the minimum and maximum angles of the polygon, and the maximum deviation the normals can have from each other. In some configurations, polygons 10759 can be filtered according to whether polygon 10759 would be too small for AV 10101 (FIG. 1A) to traverse. In some configurations, a circle the size of AV 10101 (FIG. 1A) can be dragged around each of polygons 10759 by known means. If the circle falls substantially within polygon 10759, polygon 10759, and therefore the resulting drivable surface, can accommodate AV 10101 (FIG. 1A). In some configurations, the area of polygon 10759 can be compared to the occupied area of AV 10101 (FIG. 1A). The polygon can be assumed to be irregular, and thus the first step in determining the area of polygon 10759 would be to separate polygon 10759 into regular polygons 10759A by known methods. For each regular polygon 10759A, a standard area equation can be used to determine its size. The areas of each regular polygon 10759A can be added together to determine the area of the polygon 10759, which can be compared to the occupied area of AV10101 (FIG. 1A). The filtered polygons can include a subset of polygons that meet the size criteria. The filtered polygons can be used to set the final drivable surface.

Continuing with reference to FIG. 1G, in some configurations, polygon 10759 can be processed by removing outliers by conventional means, such as, for example, but not limited to, statistical analysis techniques such as those available at the Point Cloud Library, http://pointclouds.org/documentation/tutorials/statistical_outlier.php. Filtering can include reducing the size of segment 10137 (FIG. 1D) by conventional means, including, but not limited to, a voxelized grid approach such as that available at the Point Cloud Library, http://pointclouds.org/documentation/tutorials/voxel_grid.php. The concave polygon 10263 can be created, for example, but not by way of limitation, by the process described in "A New Concave Hull Algorithm and Concaveness Measure for n-dimensional Datasets," Park et al., Journal of Information Science and Engineering 28, pp. 587-600, 2012.

Now, referring primarily to FIG. 1H, in some configurations, the processed point cloud data 10135 (FIG. 1D) can be used to determine an initial drivable surface 10265. Region growing can generate point clusters that may include points that are part of the drivable surface. In some configurations, a reference plane can be fitted to each of the point clusters to determine the initial drivable surface. In some configurations, the point clusters can be filtered according to a relationship between the orientation of the point cluster and the reference plane. For example, if the angle between the point cluster plane and the reference plane is below, for example, but not limited to, about 30°, the point cluster can be preliminarily considered to be part of the initial drivable surface. In some configurations, the point clusters can be filtered based on, for example, but not limited to, a size constraint. In some configurations, a point cluster that is larger in point size than about 20% of the total points in the point cloud data 10131 (FIG. 1D) can be considered too large, and a point cluster that is smaller in size than about 0.1% of the total points in the point cloud data 10131 (FIG. 1D) can be considered too small. The initial drivable surface can include a filtered version of the point cluster. In some configurations, the point cluster can be split for further processing by any of several known methods. In some configurations, density-based spatial clustering of noisy applications (DBSCAN) can be used to split the point cluster, while in some configurations, k-means clustering can be used to split the point cluster. DBSCAN can group points that are dense together and mark points that are substantially isolated or in low density areas as outliers. To be considered dense, a point must be located within a preselected distance from a candidate point. In some configurations, a scale factor for the preselected distance can be determined empirically or dynamically. In some configurations, the scale factor may be in the range of approximately 0.1 to 1.0.

Referring primarily to FIG. 1I, generating the SDSF line (10163) (FIG. 1D) can include locating the SDSF by further filtering of the concave polygon 10263 on the drivable surface 10265 (FIG. 1H). In some configurations, the points from the point cloud data that make up the polygon can be categorized as either upper donut points 10351 (FIG. 1J), lower donut points 10353 (FIG. 1J), or cylinder points 10355 (FIG. 1J). The upper donut points 10351 (FIG. 1J) can correspond to the shape of the SDSF model 10352 that is furthest from the ground. The lower donut points 10353 (FIG. 1J) correspond to the shape of the SDSF model 10352 that is closest to the ground or at ground level. The cylinder points 10355 (FIG. 1J) can fall into a shape between the upper donut points 10351 (FIG. 1J) and the lower donut points 10353 (FIG. 1J). A combination of categories can form a donut 10371. To determine if the donut 10371 forms an SDSF, certain criteria are tested. For example, in each donut 10371, there must be a minimum number of points that are upper donut points 10351 (FIG. 1J) and a minimum number that are lower donut points 10353 (FIG. 1J). In some configurations, the minimum value can be selected empirically and can fall into the range of about 5 to 20. Each donut 10371 can be divided into multiple parts, for example, two hemispheres. Another criterion for determining whether the points in the donut 10371 represent an SDSF is whether the majority of the points are located in opposing hemispheres of the portion of the donut 10371. The cylindrical points 10355 (FIG. 1J) can occur in either the first cylindrical region 10357 (FIG. 1J) or the second cylindrical region 10359 (FIG. 1J). Another criterion for SDSF selection is that a minimum number of points must be present in both cylindrical regions 10357/10359 (FIG. 1J). In some configurations, the minimum number of points can be selected empirically and can fall in the range of 3-20. Another criterion for SDSF selection is that the donut 10371 must include at least two of three categories of points: the upper donut points 10351 (FIG. 1J), the lower donut points 10353 (FIG. 1J), and the cylindrical points 10355 (FIG. 1J).

Continuing with reference primarily to FIG. 1I, in some configurations, polygons can be processed in parallel. Each category worker 10362 can search its assigned polygons for SDSF points 10789 (FIG. 1N) and can assign SDSF points 10789 (FIG. 1N) to categories 10763 (FIG. 1G). As polygons are processed, the resulting point categories 10763 (FIG. 1G) can be combined (10363) to form combined category 10366, and the categories can be shortened (10365) to form shortened combined category 10368. Shortening the SDSF points 10789 (FIG. 1N) can include filtering the SDSF points 10789 (FIG. 1N) with respect to their distance from the ground. The shortened combined categories 10368 can be averaged by searching the area around each SDSF point 10766 (FIG. 1G) and generating an average point 10765 (FIG. 1G), possibly processed in parallel by an average worker 10373, where the points of the categories form a set of averaged donuts 10375. In some configurations, the radius around each SDSF point 10766 (FIG. 1G) can be empirically determined. In some configurations, the radius around each SDSF point 10766 (FIG. 1G) can include a range of 0.1 m to 1.0 m. The height change between one point on the SDSF trajectory 10377 (FIG. 1G) and another for the SDSF at the average point 10765 (FIG. 1G) can be calculated. Connecting the averaged donuts 10375 together can generate the SDSF trajectory 10377 (FIGS. 1G and 1K). When creating an SDSF trajectory 10377 (FIGS. 1G and 1K), if there are two candidate next points within a search radius of the starting point, the next point can be chosen based on at least forming a straight line between the previous line segment, the starting point, and the candidate destination point as close as possible, where the candidate next point represents the smallest change in SDSF height between the previous point and the candidate next point. In some configurations, the SDSF height can be defined as the difference between the heights of the upper donut 10351 (FIG. 1J) and the lower donut 10353 (FIG. 1J).

Now referring primarily to FIG. 1L, combining concave polygons and SDSF lines 10165 (FIG. 1D) can generate a data set including polygon 10139 (FIG. 1D) as well as SDSF 10141 (FIG. 1D), which can be manipulated to generate a graphed polygon using the SDSF data. Manipulating concave polygon 10263 can include, but is not limited to, merging concave polygon 10263 to form merged polygon 10771. Merging concave polygon 10263 can be accomplished using known methods such as, for example, but not limited to, those found at (http://www.angusj.com/delphi/clipper.php). Merged polygon 10771 can be expanded to smooth edges and form expanded polygon 10772. Expanded polygon 10772 can be contracted to provide a driving margin to form contracted polygon 10774, to which SDSF trajectory 10377 (FIG. 1M) can be added. Inward trimming (contraction) can ensure that there is room near the edge for AV10101 (FIG. 1A) to travel by reducing the size of the drivable surface by at least a preselected amount based on the size of AV10101 (FIG. 1A). Polygon expansion and contraction can be accomplished by commercially available techniques such as, for example, but not limited to, the ARCGIS (registered trademark) clip command (http://desktop.arcgis.com/en/arcmap/10.3/manage-data/editing-existing-features/clipping-a-polygon-feature.htm).

Now referring primarily to FIG. 1M, the contracted polygon 10774 can be partitioned into polygons 10778, each of which can be traversed without encountering a non-drivable surface. The contracted polygon 10774 can be partitioned by conventional means such as, for example, but not limited to, ear slicing, optimized by z-order curve hashing and extended to handle holes, twisted polygons, degeneracy, and self-intersections. Commercially available ear slicing implementations can include, but are not limited to, those found at (https://github.com/mapbox/earcut.hpp). The SDSF trajectory 10377 can include SDSF points 10789 (FIG. 1N), which can be connected to the polygon vertices 10781. Vertices 10781 can be considered to be possible path points that may be connected to each other to form possible travel paths for AV 10101 (FIG. 1A). In the dataset, SDSF points 10789 (FIG. 1N) can be labeled as such. As partitioning progresses, it is possible that redundant edges are introduced, such as, for example, but not limited to, edges 10777 and 10779. Removing one of edges 10777 or 10779 can reduce the complexity of further analysis and can preserve the polygon mesh. In some configurations, a Hertel-Mehlhorn polygon partitioning algorithm can be used to remove edges and omit edges labeled as features. The set of polygons 10778, including the labeled features, can be subjected to further simplification to reduce the number of possible path points, which can be provided to the device controller 10111 (FIG. 1A) in the form of annotated point data 10379 (FIG. 5B) that can be used to populate the occupancy grid.

2A-2B, sensor data collected by the AV can also be used to populate the occupancy grid. A processor in the AV can receive data from sensors in a long-range sensor assembly 20400 mounted on the cargo container 20110, and from short-range sensors 20510, 20520, 20530, 20540, as well as other sensors located in the cargo platform 20160. In addition, the processor may receive data from an optional short-range sensor 20505 mounted near the top of the front of the cargo container 20110. The processor may also receive data from one or more antennas 20122A, 20122B (FIG. 1C), including cellular, WiFi, and/or GPS. In one embodiment, the AV 20100 has a GPS antenna 20122A (FIG. 1C) located on the long range sensor assembly 20400 and/or an antenna 20122B (FIG. 1C) located on the cargo container 20110. The processor may be located anywhere within the AV 20100. In some embodiments, one or more processors are located in the long range sensor assembly 20400. Additional processors may be located in the cargo platform 20160. In other embodiments, the processors may be located in the cargo container 20110 and/or as part of the power base 20170.

2A-2B, the long-range sensor assembly 20400 is mounted on top of the cargo container to provide an improved view of the environment surrounding the AV. In one embodiment, the long-range sensor assembly 20400 is greater than 1.2 m feet above the travel surface or ground. In other embodiments, if the cargo container is higher or the power base configuration elevates the cargo platform 20160, the long-range sensor assembly 20400 may be 1.8 m above the ground across which the AV is traveling. The long-range sensor assembly 20400 provides information about the environment around the AV from a minimum distance to a maximum range. The minimum distance may be defined by the relative positions of the long-range sensor 20400 and the cargo container 20110. The minimum distance may further be defined by the field of view (FOV) of the sensor. The maximum distance may be defined by the range of the long-range sensor in the long-range sensor assembly 20400 and/or by the processor. In one embodiment, the range of the long-range sensor is limited to 20 meters. In one embodiment, the Velodyne Puck LIDAR has a range of up to 100 m. The long-range sensor assembly 20400 may provide data about objects in all directions. The sensor assembly may provide information about structures, surfaces, and obstacles over a 360° angle around the AV20100.

Continuing with reference to FIG. 2A, three long-range cameras viewing through windows 20434, 20436, and 20438 can provide horizontal FOVs 20410, 20412, 20414 that together provide a 360° FOV. The horizontal FOV may be defined by the camera selected and the location of the camera within the long-range camera assembly 20400. In describing the field of view, a zero angle is a ray that lies in a vertical plane through the center of the AV 20100 and perpendicular to the front of the AV. A zero angle ray passes through the front of the AV. The front long-range camera viewing through window 20434 has a 96° FOV 20410 from 311° to 47°. The left long-range camera viewing through window 20436 has a FOV 20412 from 47° to 180°. The right long-range camera, viewing through window 20438, has a FOV 20414 of 180° to 311°. The long-range sensor assembly 20400 may include an industrial camera positioned to view through window 20432 that provides more detailed information about objects and surfaces in front of the AV20100 than the long-range camera. The industrial camera located behind window 20432 may have a FOV 20416 that is defined by the camera selected and the location of the camera within the long-range camera assembly 20400. In one example, the industrial camera behind window 20432 has a FOV of 23° to 337°.

Now referring to FIG. 2B, the LIDAR 20420 provides a 360° horizontal FOV around the AV 20100. The vertical FOV may be limited by the LIDAR instrument. In one example, the vertical FOV 20418 is 40° and mounted 1.2m-1.8m above the ground, setting the minimum distance of the sensor at 3.3m-5m from the AV 20100.

Now referring to Figures 2C and 2D, the long range sensor assembly 20400 is shown with a cover 20430. The cover 20430 includes windows 20434, 20432, 20436 through which the long range and industrial cameras observe the environment surrounding the AV20100. The cover 20430 for the long range sensor assembly 20400 is sealed from the weather by an O-ring between the cover 20430 and the top of the cargo container 20110.

2E and 2F, the cover 20430 has been removed to reveal an embodiment of the camera and processor. The LIDAR sensor 20420 provides data regarding range or distance to surfaces surrounding the AV. These data may be provided to the processor 20470, located in the long range sensor assembly 20400. The LIDAR is mounted on a structure 20405 above the long range cameras 20440A-C and the cover 20430. The LIDAR sensor 20420 is an example of a ranging sensor based on reflected laser pulsed light. Other ranging sensors such as radar using reflected radio waves can also be used. In one example, the LIDAR sensor 20420 is a Puck sensor by VELODYNE LIDAR® (San Jose, Calif.). The three long-range cameras 20440A, 20440B, 20440C provide digital images of objects, surfaces, and structures around the AV 20100. The three long-range cameras 20440A, 20440B, 20440C are arranged around the structure 20405 relative to the cover 20430 to provide three horizontal FOVs covering a full 360° around the AV. The long-range cameras 20440A, 20440B, 20440C are on the lift ring structure 20405, which is mounted to the cargo container 20110. The long-range cameras 20440A, 20440B, 20440C receive images through windows 20434, 20436, 20438 mounted in the cover 20430. The long-range cameras may comprise cameras and lenses on a printed circuit board (PCB).

Now referring to FIG. 2F, one embodiment of the long range camera 20440A may comprise a digital camera 20444 with a fisheye lens 20442 mounted on the front of the digital camera 20444. The fisheye lens 20442 may expand the FOV of the camera to a much wider angle. In one embodiment, the fisheye lens expands the field of view to 180°. In one embodiment, the digital camera 20444 is similar to an e-cam52A_56540_MOD by E-con Systems (San Jose, Calif.). In one embodiment, the fisheye lens 20442 is similar to a model DSL227 by Sunex (Carlsbad, Calif.).

With continued reference to FIG. 2F, the long range sensor assembly 20400 may also include an industrial camera 20450 that receives visual data through a window 20432 in the cover 20430. The industrial camera 20450 provides additional data to the processor 20470 regarding objects, surfaces, and structures in front of the AV. The camera may be similar to a Kowa industrial camera (part number LM6HC). The industrial camera 20450 and the long range cameras 20440A-C are positioned 1.2m to 1.8m above the surface over which the AV20100 is moving.

With continued reference to FIG. 2F, mounting the long-range sensor assembly 20400 on a cargo container provides at least two advantages. The field of view for the long-range sensors, including the long-range cameras 20440A-C, the industrial camera 20450, and the LIDAR 20420, is often less blocked by nearby objects such as people, cars, low walls, etc., when the sensors are mounted further above the ground. In addition, pedestrian-only roads are designed to provide visual cues, including signs, fence heights, etc., for people to perceive, with typical eye heights being in the range of 1.2m to 1.8m. Mounting the long-range sensor assembly 20400 on a cargo container places the long-range cameras 20440A-C, 20450 at the same level as the signs and above the visual cues that are directed at pedestrians. The long-range sensors are mounted on a structure 20405 that provides a substantially rigid mount to withstand deflections caused by the movement of the AV20100.

2E and 2F, the long-range sensor assembly may include an inertial measurement unit (IMU) and one or more processors that receive data from the long-range sensors and output processed data to other processors for navigation. An IMU 20460 with a vertical reference (VRU) is mounted on the structure 20405. The IMU/VRU 20460 may be located directly below the LIDAR 20420 to provide position data for the LIDAR 20420. The position and orientation from the IMU/VRU 20460 may be combined with data from other long-range sensors. In one example, the IMU/VRU 20460 is a model MTi 20 supplied by Xsens Technologies (The Netherlands). The one or more processors may include a processor 20465 that receives data from at least the industrial camera 20450. Additionally, the processor 20470 may receive data from at least one of the following: LIDAR 20420, long-range cameras 20440A-C, industrial camera 20450, and IMU/VRU 20460. The processor 20470 may be cooled by a liquid-cooled heat exchanger 20475, which is connected to a circulating coolant system.

2G, the AV 20100 may include several short-range sensors that detect driving surfaces and obstacles within a predetermined distance from the AV. The short-range sensors 20510, 20520, 20530, 20540, 20550, and 20560 are located on the periphery of the container platform 20160. These sensors are located below the cargo container 20110 (FIG. 2B) and closer to the ground than the long-range sensor assembly 20400 (FIG. 2C). The short-range sensors 20510, 20520, 20530, 20540, 20550, and 20560 are angled downward to provide a FOV that captures surfaces as well as objects that cannot be seen by the sensors in the long-range sensor assembly 20400 (FIG. 2C). The field of view of sensors located closer to the ground and angled downward is less likely to be obstructed by nearby objects and pedestrians than sensors mounted further from the ground. In one example, the short-range sensors provide information about the ground surface and objects up to 4 m from the AV20100.

2B, the vertical FOV for two of the short-range sensors is shown in a side view of the AV 20100. The vertical FOV 20542 of the rear-facing sensor 20540 is centered on a centerline 20544. The centerline 20544 is angled below the upper surface of the cargo platform 20160. In one example, the sensor 20540 has a vertical FOV of 42° and a centerline 20546 that is angled 22°-28° below a plane 20547 defined by the top plate of the cargo platform 20160. In one example, the short-range sensors 20510 and 20540 are approximately 0.55m-0.71m above the ground. The resulting FOV 20512, 20542 covers the ground 0.4m-4.2m from the AV. The short-range sensors 20510, 20520, 20530, 20550 (FIG. 2G), 20560 (FIG. 2G) mounted on the cargo base 20160 have similar vertical fields of view and centerline angles relative to the top of the cargo platform. The short-range sensors mounted on the cargo platform 20160 can see the ground 0.4 to 4.7 meters from the outer edge of the AV 20100.

With continued reference to FIG. 2B, a short-range sensor 20505 may be mounted on the front surface near the top of the cargo container 20110. In one embodiment, the sensor 20505 may provide an additional view of the ground in front of the AV to the view provided by the short-range sensor 20510. In another embodiment, the sensor 20505 may provide a view of the ground in front of the AV in place of the view provided by the short-range sensor 20510. In one embodiment, the short-range sensor 20505 may have a vertical FOV 20507 of 42°, with the angle of the centerline relative to the top of the cargo platform 20160 being 39°. The resulting view of the ground extends from 0.7m to 3.75m from the AV.

2G, the horizontal FOVs of the short-range sensors 20510, 20520, 20530, 20540, 20550, 20560 cover all directions around the AV 20100. The horizontal FOVs 20522 and 20532 of adjacent sensors such as 20520 and 20530 overlap at some distance away from the AV 20100. In one example, the horizontal FOVs 20522, 20532 and 20562, 20552 of adjacent sensors 20520, 20530 and 20560, 20550 overlap at 0.5 to 2 meters from the AV. The short-range sensors are distributed around the perimeter of the cargo base 20160, have horizontal fields of view, and are installed at specific angles to provide near complete visual coverage of the ground surrounding the AV. In one embodiment, the short range sensors have a horizontal FOV of 69°. The front sensor 20510 faces forward at a zero angle to the AV and has a FOV 20512. In one embodiment, the two front corner sensors 20520, 20560 are angled such that their centerlines are at an angle 20564 of 65°. In one embodiment, the rear sensors 20530, 20550 are angled such that the centerlines of 20530 and 20560 are at an angle 20534 of 110°. Other numbers of sensors with other horizontal FOVs are also possible, mounted around the periphery of the cargo base 20160 to provide a nearly complete view of the ground around the AV 20100 in some configurations.

Now referring to FIG. 2H, short range sensors 20510, 20520, 20530, 20540, 20550, 20560 are located on the periphery of the cargo base 20160. The short range cameras are mounted in protrusions that set the angle and location of the short range cameras. In another configuration, the sensors are mounted and positioned on the interior of the cargo base and receive visual data through windows that are aligned with the outer skin of the cargo base 20160.

2I and 2J, the short range sensor 20600 is mounted within a skin element 20516 of the cargo base 20160 and may include a liquid cooling system. The skin element 20516 includes formed protrusions 20514 that hold the short range sensor assembly 20600 at a predetermined location and vertical angle relative to the top of the cargo base 20160 and at an angle relative to the front of the cargo base 20160. In some configurations, the short range sensor 20510 is angled downward relative to the cargo platform 20160 by 28°, the short range sensors 20520 and 20560 are angled 18° downward and 25° forward, the short range sensors 20530 and 20550 are angled 34° downward and 20° aft, and the short range sensor 20540 is angled downward relative to the cargo platform 20160 by 28°. The skin element 20516 includes a cavity 20517 for receiving the camera assembly 20600. The skin element 20516 may also include a number of elements 20518 for receiving mechanical fasteners, including but not limited to rivets, screws, and buttons. Alternatively, the camera assembly may be mounted with an adhesive or held in place with clips fastened to the skin element 20516. A gasket 20519 may provide a seal against the front of the camera 20610.

2K and 2L, the short-range sensor assembly 20600 includes a short-range sensor 20610 mounted on a bracket 20622 that is attached to a water-cooled plate 20626. The outer case 20612, transparent cover 20614, and heat sink 20618 are partially removed in FIG. 2K and 2L to better visualize the heat-dissipating elements of the short-range sensor 20610, the sensor block 20616, and the electronic block 20620. The short-range sensor assembly 20600 may include one or more thermoelectric coolers (TEC) 20630 between the bracket 20622 and the liquid-cooled plate 20626. The liquid-cooled plate 20626 is cooled by coolant pumped through 20628 that is thermally connected to the plate 20626. The TEC is an electrically-driven element with first and second sides. The powered TEC cools the first side while rejecting the thermal energy removed from the first side plus electrical power on the second side. In the short range sensor assembly 20600, the TEC 20630 cools the bracket 20622 and transfers the thermal cooling energy plus electrical energy to the water plate 20626. Alternatively, the TEC 20630 can be used to actively control the temperature of the camera 20600 by varying the magnitude and polarity of the voltage supplied to the TEC 20630.

Operating the TEC 20630 in cooling mode allows the short-range sensor 20610 to operate at a temperature below the coolant temperature. The bracket 20622 is thermally connected to the short-range sensor 20610 in two locations to maximize cooling of the sensor block 20616 and the electronics block 20620. The bracket 20622 includes tabs 20624 that are thermally attached to the heat sink 20618 via screws 20625. The heat sink 20618 is thermally connected to the sensor block 20616. The bracket is thus thermally connected to the sensor block 20616 via the heat sink 20618, screws 20625, and tabs 20624. The bracket 20622 is also mechanically attached to the electronics block 20620 to provide direct cooling of the electronics block 20620. Bracket 20622 may include multiple mechanical attachments, including, but not limited to, screws and rivets that engage element 20518 of FIG. 2J. Short-range sensor 20610 may incorporate one or more sensors, including, but not limited to, a camera, a stereo camera, an ultrasonic sensor, a short-range radar, and an infrared projector and a CMOS sensor. One exemplary short-range sensor is similar to the RealSense Depth Camera D435 by Intel (Santa Clara, California), which includes an IR projector, two imager chips, and an RGB camera.

2M-2O, another embodiment of the AV20100 is shown. The AV20100A includes a cargo container 20110 mounted on a cargo platform 20160 and a power base 20170. The AV20100A includes multiple long-range and short-range sensors. A primary long-range sensor is mounted in a sensor pylon 20400A on the cargo container 20110. The sensor pylon may include a LIDAR 20420 and multiple long-range cameras (not shown) oriented in divergent directions to provide a wide field of view. In some configurations, the LIDAR 20420 can be used as described elsewhere herein to provide point cloud data that may enable, for example, but not limited to, capture of an occupancy grid, identify landmarks, provide information for locating the AV20100 within its environment, and/or determine navigable space. In some configurations, the LIDAR 20420 is mounted in a sensor pylon 20400A mounted on a cargo platform 20160 and a power base 20170. The AV20100A includes multiple long-range and short-range sensors. A primary long-range sensor is mounted in a sensor pylon 20400A on the cargo container 20110. The sensor pylon may include a LIDAR 20420 and multiple long-range cameras (not shown) oriented in divergent directions to provide a wide field of view. In some configurations, the LIDAR 20420 can be used as described elsewhere herein to provide point cloud data that may enable, for example, but not limited to, capture of an occupancy grid, identify landmarks, provide information for locating the AV20100 within its environment, and/or provide information for determining navigable space. In some configurations, the LIDAR 20420 is mounted in a sensor pylon 20400A mounted on a cargo platform 20160 and a power base 20170. A long-range camera from the can be used to identify landmarks, locate the AV20100 within its environment, and/or determine navigable space.

With continued reference to Figures 2M-2O, short-range sensors are primarily mounted within the cargo platform 20160 and provide information about obstacles in the vicinity of the AV20100A. In some embodiments, the short-range sensors provide data about obstacles and surfaces within 4m of the AV20100A. In some configurations, the short-range sensors provide information up to 10m from the AV20100A. Multiple cameras, at least partially forward-facing, are mounted within the cargo platform 20160. In some configurations, the multiple cameras may include three cameras.

2O, the top cover 20830 is partially cut away to reveal the sub-roof 20810. The sub-roof 20810 provides a single piece on which multiple antennas 20820 may be mounted. In one embodiment, ten antennas 20820 are mounted on the sub-roof 20810. Additionally, there are, for example, four cellular communication channels each having two antennas, and two WiFi antennas. The antennas are wired as primary antennas for cellular transmission and reception, as well as auxiliary antennas. The auxiliary antennas may improve cellular functionality in several ways, including, but not limited to, reducing interference and achieving 4G LTE connectivity. The sub-roof 20810 as well as the top cover 20830 are non-metallic. The subroof 20810 is a plastic surface within 10-20 mm of the top cover 20830 that is non-structural and allows the antenna to be connected to the processor before the top cover 20830 is attached. The antenna connections are often high impedance and sensitive to dirt, grease, and misuse. Mounting and connecting the antenna to the subroof 20810 allows the top cover to be installed and removed without touching the antenna connections. Maintenance and repair operations may include removing the top cover without removing the subroof or disconnecting the antenna. Assembly of the antenna separate from installation of the top cover 20830 facilitates testing/repair. The top cover 20830 is weatherproof and prevents water and debris from entering the cargo container 20110. Mounting the antenna on the subroof minimizes the number of openings on the top cover 20830.

Now referring to Figures 2P, 2Q, and 2R, another embodiment of a long range sensor assembly (LRSA) 20400A is shown mounted on a cargo container (not shown). The LRSA may include a LIDAR and multiple long range cameras mounted at different locations on the LRSA structure 20950 to provide a panoramic view of the AV20100A's environment. The LIDAR 20420 is mounted at the top on the LRSA structure 20950 to provide an uninterrupted view. The LIDAR may include a VELODYNE LIDAR. Multiple long range cameras 20910A-20910D are mounted on the LRSA structure 20950 on the next level below the LIDAR 20420. In one embodiment, four cameras are mounted, one every 90° around the perimeter of the structure, providing four views of the AV20100's surrounding environment. In some embodiments, the four views will overlap. In some embodiments, each camera is either aligned with the direction of motion or orthogonal to the direction of movement. In one embodiment, one camera is aligned with each of the major faces of the AV20100A, i.e., front, back, left, and right. In one embodiment, the long-range camera is a model LI-AR01 44-MIPI-M12 made by Leopard Imaging Inc. The long-range camera may have a MIPI CSI-2 interface to provide high-speed data transfer to the processor. The long-range camera may have a horizontal field of view of 50° to 70° and a vertical field of view of 30° to 40°.

Now referring to Figures 2S and 2T, the long range processor 20940 is located on the LRSA structure 20950 below the long range cameras 20910A-20910D and the LIDAR 20420. The long range processor 20940 receives data from the long range cameras and the LIDAR. The long range processor communicates with one or more processors elsewhere in the AV20100A. The long range processor 20940 provides data derived from the long range cameras and the LIDAR to one or more processors elsewhere on the AV20100A, described elsewhere herein. The long range processor 20940 may be liquid cooled by a cooler 20930. The cooler 20930 may be mounted on a structure below the long range cameras and the LIDAR. The cooler 20930 may provide a mounting location for the long range processor 20940. The cooler 20930 is described in U.S. Patent Application Serial No. 16/883,668, filed May 26, 2020, and entitled "Apparatus for Electronic Cooling on an Autonomous Device," Attorney Docket No. AA280, which is incorporated herein by reference in its entirety. The cooler is provided with a liquid supply conduit and a return conduit that provide cooling liquid to the cooler 20930.

Referring again to Figures 2M and 2N, short-range camera assemblies 20740A-C are mounted on the front of the container platform 20160 and angled to collect information about the travel surface and obstacles, steps, curbs, and other substantially discontinuous surface features (SDSF). The camera assemblies 20740A-C include one or more LED lights to illuminate the travel surface, objects on the ground, and SDSF.

Now referring to Figures 2U-2X, camera assemblies 20740A-B include lights 20732 to illuminate the ground and objects and provide improved image data from camera 20732. Note that camera assembly 20740A is a mirror image of 20740C and the description of 20740A implicitly applies to 20740C. Camera 20732 may include a single focus camera, a stereo camera, and/or an infrared projector and a CMOS sensor. One example of a camera is the Real-Sense Depth D435 camera by Intel (Santa Clara, CA), which includes an IR projector, two imager chips with lenses, and an RGB camera. LED lights 20734 may be used at night or in low light conditions, or may be used full time to improve image data. One operating principle is that the light creates contrast by illuminating the projection surface and creating shadows in recesses. The LED lights may be white LEDs in one embodiment. In one embodiment, the LED lights 20734 are Xlamp XHP50s from Cree Inc. In another embodiment, the LED lights may emit infrared light to provide illumination for the camera 20372 without distracting or disturbing nearby pedestrians or drivers.

Continuing to refer to Figures 2U-2X, the placement and angle of the light 20374 and the shape of the covers 20736A, 20736B prevent the camera 20732 from seeing the light 20374. The angle and placement of the light 20374 and covers 20736A, 20736B prevent the light from interfering with drivers or obstructing pedestrians. It is advantageous for the camera 20732 not to be exposed to the light 20734 to prevent the sensor in the camera 20732 from being blinded by the light 20734 and thus prevented from detecting weaker light signals from the ground and objects in front and to the sides of the AV20100A. The camera 20732 and/or light 20734 may be cooled using a liquid flowing in and out of the camera assembly through the port 20736.

Now referring to Figures 2W and 2X, the short-range camera assembly 20740A includes an ultrasonic or sonar short-range sensor 20730A. The second short-range camera assembly 20740C also includes an ultrasonic short-range sensor 20730B (Figure 2N).

Now referring to FIG. 2Y, the ultrasonic sensor 20730A is mounted above the camera 20732. In one embodiment, the centerline of the ultrasonic sensor 20730A is parallel to the base of the cargo container 20110, which often means that the sensor 20730A is horizontal. The sensor 20730A is angled 45° from facing forward. The cover 20376A provides a horn 20746 for directing the ultrasonic waves emerging from and received by the ultrasonic sensor 20730A.

Continuing with FIG. 2Y, a cross section of the camera 20732, light 20734 in camera assembly 20740A illustrates the angles and openings in cover 20736. The cameras in the short-range camera assembly are angled downward to better image the ground in front and to the sides of the AV. The center camera assembly 20740B is oriented directly ahead in the horizontal plane. The corner camera assemblies 20740A, 20740C are angled 25° to their respective sides relative to directly ahead in the horizontal plane. The camera 20732 is angled 20° downward relative to the top of the cargo platform in the vertical plane. Since the AV generally holds the cargo platform horizontal, the cameras are therefore angled 20° downward from horizontal. Similarly, the center camera assembly 20740B (FIG. 2M) is angled 28° downward from horizontal. In one embodiment, the cameras in the camera assemblies may be angled downward by 25° to 35°. In another embodiment, the cameras in the camera assemblies may be angled downward by 15° to 45°. The LED lights 20734 are similarly angled downward to illuminate the ground imaged by the camera 20732 and minimize distraction to pedestrians. In one embodiment, the LED light centerline 20742 is parallel to and within 5° of the camera centerline 20738. The cover 20736A protects both the camera 20732 and pedestrians from the bright light of the LEDs 20734 in the camera assemblies 20740A-C. The cover, which isolates the light emitted by the LEDs, also provides a tapered opening 20737 to maximize the field of view of the camera 20732. The lights are recessed at least 4 mm from the opening in the cover. The light opening is defined by an upper wall 20739 and a lower wall 20744. The upper wall 20739 is approximately parallel (±5°) to the center line 2074. The lower wall 20744 diverges approximately 18° from the center line 20742 to maximize illumination of the ground and objects near the ground.

2Z-2AA, in one configuration, the light 20734 includes two LEDs 20734A each under a square lens 20734B to generate a beam of light. The LEDs/lenses are angled and positioned relative to the camera 20372 to illuminate the camera's field of view with little light spill outside the camera's 20732 FOV. The two LEDs/lenses are mounted together on a single PCB 20752 with a defined angle 20762 between the two lights. In another configuration, the two LEDs/lenses are individually mounted on a heat sink 20626A on a separate PCB at an angle to each other. In one embodiment, the light is an Xlamp XHP50 from Cree, Inc. and the lens is a 60° lens HB-SQ-W from LEDil. The lights are angled at approximately 50° to each other so the angle 20762 between the front of the lenses is 130°. The lights are located approximately 18 mm ( ± 5 mm) 20764 behind the front of the camera 20732 and approximately 30 mm 20766 below the centerline of the camera 20732.

Now referring to Figures 2AA-2BB, the camera 20732 is cooled by a thermoelectric cooler (TEC) 20630, which is cooled along with the light 20734 by liquid coolant flowing through the cold block 20626A. The camera is attached to the bracket 20622 via a screw 20625 that threads into the sensor block portion of the camera, while the rear of the bracket 20622 is bolted to the electronics block of the camera. The bracket 20622 is cooled by two TECs to maintain the performance of the IR imaging chip (a CMOS chip) in the camera 20732. The TECs reject heat from the bracket 20622 and the power they draw to the cold block 20626A.

Referring now to FIG. 2BB, the coolant is directed through a U-shaped path created by center fin 20626D. The coolant flows directly behind the LED/lens/PCB of light 20734. Fins 20626B, 20626C improve heat transfer from light 20734 to the coolant. The coolant flows upward and passes by the hot side of TEC 20630. The fluid path is created by plate 20737 (FIG. 2X) that is attached to the rear of cold block 20626A.

Now referring to FIG. 3A, sensor data and map data can be used to update an occupancy grid. The systems and methods of the present teachings can manage a global occupancy grid for a device navigating autonomously with respect to a grid map. The grid map can include routes or paths that the device may follow from a start point to a destination. The global occupancy grid can include free space indications that can indicate where it is safe for the device to navigate. The possible paths and free space indications can be combined on the global occupancy grid to establish an optimal path on which the device may proceed to safely reach the destination.

Continuing with reference to FIG. 3A, as the device moves, a global occupancy grid that would be used to determine a non-obstructive navigation route can be accessed based on the location of the device, and the global occupancy grid can be updated as the device moves. The updates can be based on at least current values associated with the global occupancy grid at the location of the device, a static occupancy grid that can include historical information about the neighborhood in which the device is navigating, and data collected by sensors as the device progresses. The sensors can be located on the device as described herein, and they can be located at different locations.

Continuing with still further reference to FIG. 3A, the global occupancy grid can include cells, which can be associated with occupancy probability values. Each cell of the global occupancy grid can be associated with information such as whether an obstacle has been identified at the cell's location, characteristics and discontinuities of the traveling surface at and surrounding the location, as determined from previously collected data and by data collected as the device navigates, and prior occupancy data associated with the location. Data captured as the device navigates can be stored in a local occupancy grid whose center is the device. When updating the global occupancy grid, the static previously collected data can be combined with the local occupancy grid data and global occupancy data determined in previous updates to create a new global occupancy grid with the space occupied by the device marked as unoccupied. In some configurations, a Bayesian method can be used to update the global occupancy grid. The method can include, for each cell in the local occupancy grid, calculating a location of the cell on a global occupancy grid, accessing a value at that location from a current global occupancy grid, accessing a value at that location from a static occupancy grid, accessing a value at that location from the local occupancy grid, and calculating a new value at that location on the global occupancy grid as a function of the current value from the global occupancy grid, the value from the static occupancy grid, and the value from the local occupancy grid. In some configurations, the relationship used to calculate the new value can include the sum of the static value and the local occupancy grid value minus the current value. In some configurations, the new value can be limited by a preselected value, for example, based on a computational limit.

Continuing with reference to FIG. 3A, the system 30100 of the present teachings can manage a global occupancy grid. The global occupancy grid can start with initial data and can be updated as the device moves. Creating the initial global occupancy grid can include a first process and updating the global occupancy grid can include a second process. The system 30100 can include, but is not limited to, a global occupancy server 30121 that can receive information from various sources and update the global occupancy grid 30505 based at least on the information. The information can be provided by, for example, but not limited to, sensors located on the device and/or elsewhere, static information, and navigation information. In some configurations, the sensors can include cameras and radars that can detect, for example, surface characteristics and obstacles. The sensors can be advantageously located on the device to provide sufficient surrounding coverage to enable safe navigation by the device, for example. In some configurations, the LIDAR 30103 can provide LIDAR point cloud (PC) data 30201, which can enable populating a local occupancy grid with LIDAR free space information 30213. In some configurations, a conventional ground detection inverse sensor model (ISM) 30113 can process the LIDAR PC data 30201 and generate the LIDAR free space information 30213.

Continuing with reference to FIG. 3A, in some configurations, an RGB-D camera 30101 can provide RGB-D PC data 30202 and RGB camera data 30203. The RGB-D PC data 30202 can populate a local occupancy grid with depth free-space information 30209, and the RGB-D camera data 30203 can populate a local occupancy grid with surface data 30211. In some configurations, the RGB-D PC data 30202 can be processed by, for example, but not limited to, a conventional stereo free-space ISM 30109, and the RGB-D camera data 30203 can be fed into, for example, but not limited to, a conventional surface detection neural network 30111. In some configurations, an RGB MIPI camera 30105 provides RGB data 30205 that can be combined with the LIDAR PC data 30201 to generate a local occupancy grid with LIDAR/MIPI free space information 30215. In some configurations, the RGB data 30205 can be fed into a conventional free space neural network 30115, the output of which, along with the LIDAR PC data 30201, can undergo a pre-selected mask 30221 that can identify the portions of the RGB data 30205 that are most important for accuracy before being fed into a conventional 2D-3D registration 30117. The 2D-3D registration 30117 can project an image from the RGB data 30205 onto the LIDAR PC data 30201. In some configurations, the 2D-3D registration 30117 is not required. Any combination of sensors and methods for processing sensor data can be used to accumulate data and update the global occupancy grid. Any number of free space estimation procedures can be used and combined to enable determination and validation of occupancy probabilities within the global occupancy grid.

Continuing with reference to FIG. 3A, in some configurations, historical data can be provided by a repository 30107 of previously collected and processed data, for example, with information associated with the navigation area. In some configurations, the repository 30107 can include route information, for example, but not limited to, polygons 30207. In some configurations, these data can be fed into a conventional polygon parser 30119, which can provide edges 30303, discontinuities 30503, and surfaces 30241 to a global occupancy grid server 30121. The global occupancy grid server 30121 can fuse local occupancy grid data collected by sensors with the processed repository data to determine a global occupancy grid 30505. A grid map 30601 (FIG. 3D) can be created from the global occupancy data.

Now referring to FIG. 3B, in some configurations, the sensor may include a sonar 30141 that may provide a local occupancy grid with sonar free space 30225 to a global occupancy grid server 30121. The depth data 30209 may be processed by a conventional free space ISM 30143. The local occupancy grid with sonar free space 30225 may be fused with the local occupancy grid with surfaces and discontinuities 30223, the local occupancy grid with LIDAR free space 30213, the local occupancy grid with LIDAR/MIPI free space 30215, the local occupancy grid with stereo free space 30209, as well as edges 30303 (FIG. 3F), discontinuities 30503 (FIG. 3F), navigation points 30501 (FIG. 3F), surface confidence 30513 (FIG. 3F), and surfaces 30241 (FIG. 3F) to form a global occupancy grid 30505.

3C-3F, to initialize a global occupancy grid, global occupancy grid initialization 30200 can include creating, by global occupancy grid server 30121, a global occupancy grid 30505 and a static grid 30249. Global occupancy grid 30505 can be created by blending data from local occupancy grid 30118 with edges 30303, discontinuities 30503, and surfaces 30241 located within the region of interest. Static grid 30249 (FIG. 3D) can be created to include data such as, for example, but not limited to, surface data 30241, discontinuities data 30503, edges 30303, and polygons 30207. An initial global occupancy grid 30505 can be calculated by adding occupancy probability data from static grid 30249 (FIG. 3E) to occupancy data derived from collected data from sensor 30107A and subtracting occupancy data from a prior 30505A (FIG. 3F) of global occupancy grid 30505. Local occupancy grid 30118 can include, but is not limited to, local occupancy grid data resulting from stereo free space estimation 30209 (FIG. 3B) through ISM, local occupancy grid data including surface/discontinuity detection results 30223 (FIG. 3B), local occupancy grid data resulting from LIDAR free space estimation 30213 (FIG. 3B) through ISM, and in some configurations, local occupancy grid data resulting from LIDAR/MIPI free space estimation 30215 (FIG. 3B) following 2D-3D registration 30117 (FIG. 3B). In some configurations, the local occupancy grid 30118 can include local occupancy grid data coming from the sonar free space estimate 30225 through the ISM. In some configurations, the various local occupancy grids with the free space estimates can be fused into the local occupancy grid 30118 according to a preselected known process. From the global occupancy grid 30505, a grid map 30601 (FIG. 3E) can be created that can include occupancy and surface data in the vicinity of the device. In some configurations, the grid map 30601 (FIG. 3E) and the static grid 30249 (FIG. 3D) can be published using, for example, but not limited to, the robotic motion system (ROS) subscribe/publish feature.

3G and 3H, to update the occupancy grid as the device moves, the occupancy grid update 30300 can include updating the local occupancy grid with data measured as the device is moving and combining those data with the static grid 30249. The static grid 30249 is accessed when the device moves out of the occupancy grid range in which it is working. The device can be positioned in the occupancy grid 30245A at a first location 30513A at a first time. When the device moves to a second location 30513B, the device is positioned in the occupancy grid 30245B, which includes a set of values derived from its new location and possibly from the values in the occupancy grid 30245A. Surface data from the initial global occupancy grid 30505 (FIG. 3C), which is geographically located with the cells in the occupancy grid 30245B at the second time and data from the static grid 30249, can be used with the measured surface data and occupancy probabilities to update each grid cell according to a preselected relationship. In some configurations, the relationship can include summing static data with measured data. The resulting occupancy grid 30245C at the third time and third location 30513C can be made available to the movement manager 30123 to inform navigation of the device.

3I, a method 30450 for creating and managing an occupancy grid can include, but is not limited to, converting sensor measurements to a reference frame associated with the device by a local occupancy grid creation node 30122, creating a time-stamped measurement occupancy grid, and publishing the time-stamped measurement occupancy grid as a local occupancy grid 30234 (FIG. 3G) 30455. A system associated with the method 30450 can include multiple local grid creation nodes 30122, e.g., one per sensor, such that multiple local occupancy grids 30234 (FIG. 3G) can be produced. The sensors can include, but are not limited to, an RGB-D camera 30325 (FIG. 3G), a LIDAR/MIPI 30231 (FIG. 3G), and a LIDAR 30233 (FIG. 3G). A system associated with the method 30450 may include a global occupancy grid server 30121 that may receive the local occupancy grids and process them according to the method 30450. In particular, the method 30450 may include loading 30242 surfaces, accessing 30504 surface discontinuities such as, but not limited to, curbs, and creating 30248 a static occupancy grid 30249 from any properties available in the repository 30107, which may include, but are not limited to, surfaces and surface discontinuities. The method 30450 may include receiving 30456 a published local occupancy grid, and moving 30457 the global occupancy grid to keep the device at the center of the map. The method 30450 may include setting 30459 a new area on the map with prior information from the static pre-occupancy grid 30249, and marking 30461 an area currently occupied by the device as unoccupied. The method 30450 may perform a loop 30463 for each cell in each local occupancy grid. The loop 30463 may include, but is not limited to, calculating a location of the cell on the global occupancy grid, accessing a previous value at that location on the global occupancy grid, and calculating a new value at the cell location based on a relationship between the previous value and the value at the cell in the local occupancy grid. The relationship may include, but is not limited to, summing the values. The loop 30463 may include comparing the new value against a preselected acceptable probability range and setting the global occupancy grid with the new value. The comparison may include setting the probability to the minimum or maximum acceptable probability if the probability is lower or higher than the minimum or maximum acceptable probability. The method 30450 may include publishing 30467 the global occupancy grid.

Now referring to FIG. 3J, an alternative method 30150 for creating a global occupancy grid may include, but is not limited to, in 30151, if the device has moved, accessing occupancy probability values associated with the old map area (where the device was before it moved) and updating the global occupancy grid on the new map area (where the device was after it moved) with the values from the old map area 30153, accessing drivable surfaces associated with cells of the global occupancy grid in the new map area and updating the cells in the updated global occupancy grid with the drivable surfaces 30155, and proceeding to step 30159. If the device has not moved at 30151 and the global occupancy grid is co-located with the local occupancy grid, the method 30150 may include updating the possibly updated global occupancy grid with surface confidences associated with the drivable surfaces from at least one local occupancy grid 30159, updating the updated global occupancy grid with log odds of occupancy probability values from at least one local occupancy grid, for example, but not limited to, using a Bayesian function 30161, and adjusting the log odds based on at least a characteristic associated with the location 30163. If the global occupancy grid is not co-located with the local occupancy grid at 30157, the method 30150 may include returning to step 30151. The characteristic may include, but is not limited to, setting the location of the device as unoccupied.

Now referring to FIG. 3K, in another configuration, a method 30250 for creating a global occupancy grid can include, but is not limited to, at 30251, if the device is moved, updating 30253 the global occupancy grid with information from a static grid associated with the new location of the device. The method 30250 can include analyzing 30257 the surface at the new location. At 30259, if the surface is drivable, the method 30250 can include updating 30261 the surface on the global occupancy grid and updating 30263 the global occupancy grid with values from a repository of static values associated with the new location on the map.

3L, updating the surface 30261 may include, but is not limited to, accessing 30351 a local occupancy grid (LOG) for a particular sensor. At 30353, if there are additional cells in the local occupancy grid to process, the method 30261 may include accessing 30355 a surface classification confidence value and a surface classification from the local occupancy grid. At 30357, if the surface classification at the cell in the local occupancy grid is identical to the surface classification in the global occupancy grid at the location of the cell, the method 30261 may include setting 30461 a new global occupancy grid (GOG) surface confidence to the sum of the old global occupancy grid surface confidence and the local occupancy grid surface confidence. At 30357, if the surface classification at the cell in the local occupancy grid is not identical to the surface classification in the global occupancy grid at the location of the cell, the method 30261 may include setting 30359 a new global occupancy grid surface confidence to the difference between the old global occupancy grid surface confidence and the local occupancy grid surface confidence. In 30463, if the new global occupancy grid surface confidence is less than zero, the method 30261 may include setting the new global occupancy grid surface classification to the value of the local occupancy grid surface classification 30469.

3M, updating the global occupancy grid with values from the repository of static values 30263 can include, but are not limited to, the following: if there are more cells to process in the local occupancy grid at 30361, the method 30263 can include accessing the log odds from the local occupancy grid 30363 and updating the log odds in the global occupancy grid with the value from the local occupancy grid at that location 30365. If the maximum certainty that the cell is empty is met at 30367, and if the device is proceeding within a given lane barrier at 30369, and if the surface is drivable at 30371, the method 30263 can include updating the probability that the cell is occupied 30373 and returning to continue processing more cells. If the maximum certainty that the cell is empty is not reached at 30367, or if the device is not traveling in a given lane at 30369, or if the surface is not drivable in the mode the device is currently traveling in at 30371, method 30263 may include returning to consider additional cells without updating the log-odds. If the device is in a standard mode, i.e., a mode in which the device may navigate relatively uniform surfaces, and the surface classification indicates that the surface is relatively non-uniform, method 30263 may adjust the device's path by increasing the probability that the cell is occupied by updating 30373 the log-odds. If the device is in a standard mode and the surface classification indicates that the surface is relatively uniform, method 30263 may adjust the device's path by decreasing the probability that the cell is occupied by updating 30373 the log-odds. If the device is traveling in four-wheel mode, i.e., a mode in which the device can navigate uneven terrain, an adjustment to the probability that the cell is occupied may not be necessary.

4A, the AV can proceed in a specific mode that may be associated with a device configuration, for example, the configuration depicted in device 42114A and the configuration depicted in device 42114B. A system of the present teachings for real-time control of the configuration of the device based on at least one environmental factor and the status of the device can include, without limitation, a sensor, a means of locomotion, a chassis operatively coupled to the sensor and the means of locomotion, the chassis being driven by a motor and a power supply, a device processor that receives data from the sensor, and a power base processor that controls the means of locomotion. In some configurations, the device processor can receive environmental data, determine environmental factors, determine configuration changes according to the environmental factors and the status of the device, and provide the configuration changes to the power base processor. The power base processor can issue commands to the means of locomotion to move the device from location to location and physically reconfigure the device when required by road surface type.

Continuing with reference to FIG. 4A, sensors that collect environmental data can include, for example, but are not limited to, cameras, LIDAR, radar, thermometers, pressure sensors, and weather condition sensors, some of which are described herein. From this set of data, the device processor can determine environmental factors upon which device configuration changes can be based. In some configurations, the environmental factors can include surface factors, such as, for example, but not limited to, surface type, surface characteristics, and surface conditions. The device processor can determine in real time how to change the configuration to accommodate traversing the detected surface type based on the environmental factors and the current status of the device.

Continuing with reference to FIG. 4A, in some configurations, the configuration change of the devices 42114A/B/C (collectively referred to as the devices 42114) can include, for example, a change in the configuration of the means of locomotion. Other configuration changes, such as user information displays and sensor controls, that may depend on the current mode and surface type, are also envisioned. In some configurations, the means of locomotion can include at least four drive wheels 442101, two on each side of the chassis 42112, and at least two caster wheels 42103 operably coupled to the chassis 42112, as described herein. In some configurations, the drive wheels 442101 can be operably coupled in pairs 42105, each pair 42105 can include a first drive wheel 42101A and a second drive wheel 42101B of the four drive wheels 442101, the pairs 42105 being located on opposite sides of the chassis 42112, respectively. The operable coupling can include a wheel cluster assembly 42110. In some configurations, a power base processor 41016 (FIG. 4B) can control the rotation of the cluster assembly 42110. Left and right wheel motors 41017 (FIG. 4B) can drive the wheels 442101 on either side of the chassis 42112. Turning can be accomplished by driving the left and right wheel motors 41017 (FIG. 4B) at different rates. A cluster motor 41019 (FIG. 4B) can rotate the wheel base in a forward/rearward direction. The rotation of the wheel base can allow the cargo to rotate independently of the drive wheels 442101, if at all, while the front drive wheels 442101A are higher or lower than the rear drive wheels 442101B, for example, when encountering a discontinuous surface feature. The cluster assembly 42110 can operate each pair of two wheels 42105 independently, thereby providing forward, reverse, and rotational motion of the device 42114 upon command. The cluster assembly 42110 can provide structural support for the pairs 42105. The cluster assembly 42110 can provide mechanical power to rotate the wheel drive assemblies together, enabling functions that depend on cluster assembly rotation, such as, but not limited to, climbing discontinuous surface features, various surface types, and uneven terrain. Further details on the operation of the clustered wheels can be found in U.S. Patent Application No. 16,035,205 (Attorney Docket No. X80), filed July 13, 2018, and entitled "Mobility Device," which is incorporated herein by reference in its entirety.

Continuing with reference to FIG. 4A, the configuration of device 42114 can be associated, including but not limited to, with mode 41033 (FIG. 4B) of device 42114. Device 42114 can operate in several of modes 41033 (FIG. 4B). In standard mode 10100-1 (FIG. 5E), device 42114B can operate on two of drive wheels 442101B and two of caster wheels 42103. Standard mode 10100-1 (FIG. 5E) can provide turning capability and mobility on relatively firm, level surfaces, such as, but not limited to, indoor environments, sidewalks, and pavements. In the enhanced mode 10100-2 (FIG. 5E) or four-wheel mode, the device 42114A/C can command four of the drive wheels 442101A/B and can be actively stabilized through on-board sensors to raise and/or reorient the chassis 42112, casters 42103, and cargo. The four-wheel mode 10100-2 (FIG. 5E) can provide mobility in a variety of environments and enable the device 42114A/C to climb steep inclines and navigate across soft, uneven terrain. In the four-wheel mode 10100-2 (FIG. 5E), all four of the drive wheels 442101A/B can be deployed and the caster wheels 42103 can be retracted. Rotation of the cluster 42110 can enable operation on uneven terrain and the drive wheels 442101A/B can run over and across discontinuous surface features. This functionality can provide the device 42114A/C with mobility in a wide variety of outdoor environments. The device 42114B can operate on outdoor surfaces that are firm and stable, but wet. Frost heave and other natural phenomena can degrade outdoor surfaces, causing cracks and loose material. In the four-wheel mode 10100-2 (FIG. 5E), the device 42114A/C can operate on these degraded surfaces. The mode 41033 (FIG. 4B) is described in detail in U.S. Pat. No. 6,571,892 ('892), issued Jun. 3, 2003, and entitled "Control System and Method," which is incorporated herein by reference in its entirety.

4B, the system 41000 can drive the device 42114 (FIG. 4A) by processing inputs from the sensor 41031 and generating commands to the wheel motor 41017 to drive the wheel 442101 (FIG. 4A) and to the cluster motor 41019 to drive the cluster 42110 (FIG. 4A). The system 41000 can include, but is not limited to, a device processor 41014 and a power base processor 41016. The device processor 41014 can receive and process environmental data 41022 from the sensor 41031 and provide configuration information 40125 to the power base processor 41016. In some configurations, the device processor 41014 can include a sensor processor 41021, which can receive and process environmental data 41022 from the sensor 41031. The sensor 41031 can include, but is not limited to, a camera, as described herein. From these data, information about the driving surface being traversed, for example, by the device 42114 (FIG. 4A) can be accumulated and processed. In some configurations, the driving surface information can be processed in real-time. The device processor 41014 can include a configuration processor 41023 that can determine, for example, a surface type 40121 from the environmental data 41022 that is being traversed by the device 42114 (FIG. 4A). The configuration processor 41023 can include, for example, a driving surface processor 41029 (FIG. 4C) that can create, for example, a driving surface classification layer, a driving surface confidence layer, and an occupancy layer from the environmental data 41022. These data can be used by the power base processor 41016 to create movement commands 40127 and motor commands 40128 as described herein, and by the global occupancy grid processor 41025 to update an occupancy grid that can be used for path planning. The configuration 40125 can be based, at least in part, on the surface type 40121. The surface type 40121 and mode 41033 can be used to determine occupancy grid information 41022, which can include, at least in part, the probability that a cell in the occupancy grid is occupied. The occupancy grid can, at least in part, enable the determination of possible paths that the device 42114 (FIG. 4A) can take.

Continuing with reference to FIG. 4B, the power base processor 41016 can receive configuration information 40125 from the device processor 41014 and process the configuration information 40125 along with other information, e.g., routing information. The power base processor 41016 can include a control processor 40325 that can create movement commands 40127 based on at least the configuration information 40125 and provide the movement commands 40127 to the motor drive processor 40326. The motor drive processor 40326 can generate motor commands 40128 that can direct and move the device 42114 (FIG. 4A). Specifically, the motor drive processor 40326 can generate motor commands 40128 that can drive the wheel motors 41017 and can generate motor commands 40128 that can drive the cluster motors 41019.

4C, the real-time surface detection of the present teachings can include a configuration processor 41023, which can include, but is not limited to, a travel surface processor 41029. The travel surface processor 41029 can determine characteristics of the travel surface over which the device 42114 (FIG. 4A) is navigating. The characteristics can be used to determine the future configuration of the device 42114 (FIG. 4A). The travel surface processor 41029 can include, but is not limited to, a neural network processor 40207, data conversion 40215, 40219, and 40239, a layer processor 40241, and an occupancy grid processor 40242. Together, these components can generate information that can direct changes in the configuration of the device 42114 (FIG. 4A) and can enable modifications to the occupancy grid 40244 (FIG. 4C) that can inform the path planning for the progression of the device 42114 (FIG. 4A).

4C and 4D, the neural network processor 40207 can subject the environmental data 41022 (FIG. 4B) to a trained neural network that can indicate, for each point of data collected by the sensor 41031 (FIG. 4B), the type of surface that the point is likely to represent. The environmental data 41022 (FIG. 4B) can be received as, without limitation, a camera image 40202, where the camera can be associated with a camera property 40204. The camera image 40202 can include a 2D grid of points 40201 having an X-resolution 40205 (FIG. 4D) and a Y-resolution 40204 (FIG. 4D). In some configurations, the camera image 40202 can include an RGB-D image, where the X-resolution 40205 (FIG. 4D) can include 40,640 pixels and the Y-resolution 40204 (FIG. 4D) can include 40,480 pixels. In some configurations, the camera images 40202 can be converted into images that are formatted according to the requirements of the selected neural network. In some configurations, the data can be normalized, scaled, and converted from 2D to 1D, which can improve the processing efficiency of the neural network. The neural network can be trained in a number of ways, including, but not limited to, training with RGB-D camera images. In some configurations, the trained neural network, represented in a neural network file 40209 (FIG. 4D), can be made available to the neural network processor 40207 through a direct connection to a processor executing the trained neural network, or through, for example, a communication channel. In some configurations, the neural network processor 40207 can use the trained neural network file 40209 (FIG. 4D) to identify surface types 40121 (FIG. 4C) in the environmental data 41022 (FIG. 4B). In some configurations, the surface types 40121 (FIG. 4C) can include, but are not limited to, non-driveable, hard driveable, soft driveable, and curb. In some configurations, the surface types 40121 (FIG. 4C) can include, but are not limited to, non-driveable/background, asphalt, concrete, brick, packed dirt, wood plank, gravel/pebbles, grass, mulch, sand, curb, solid metal, metal grid, tactile paving, snow/ice, and railroad tracks. The result of the neural network processing can include a surface classification grid 40303 of points 40213 having an X resolution 40205, a Y resolution 40203, and a center 40211. Each point 40213 in the surface classification grid 40303 can be associated with a likelihood of being a specific one of the surface types 40121 (FIG. 4C).

Continuing with reference to Figures 4C and 4D, the travel surface processor 41029 (Figure 4C) may include a 2D-to-3D transformation 40215 that may backproject from the 2D surface classification grid 40303 (Figure 4D) in the 2D camera frame to a 3D image cube 40307 (Figure 4D) in 3D real world coordinates as seen by the camera. The backprojection may restore the 3D nature of the 2D data and transform the 2D image from the RGB-D camera into the 3D camera frame 40305 (Figure 4C). Points 40233 (Figure 4D) in the cube 40307 (Figure 4D) may each be associated with a likelihood of being a particular one of the surface types 40121 (Figure 4C), and a depth coordinate and an X/Y coordinate. The dimensions of the point cube 40307 (FIG. 4D) can be bounded according to the camera properties 40204 (FIG. 4C), such as, for example, but not limited to, focal length x, focal length y, and center of projection 40225. For example, the camera properties 40204 can include a maximum range over which the camera can reliably project. Additionally, there may be features of the device 42114 (FIG. 4A) that can interfere with the image 40202. For example, the caster 42103 (FIG. 4A) may interfere with the view of the camera 40227 (FIG. 4D). These factors can limit the number of points in the point cube 40307 (FIG. 4D). In some configurations, the camera 40227 (FIG. 4D) cannot reliably project beyond about 6 meters, which can represent an upper limit for the range of the camera 40227 (FIG. 4D), and can limit the number of points in the point cube 40307 (FIG. 4D). In some configurations, features of device 42114 (FIG. 4A) can act as a minimum limit for the range of camera 40227 (FIG. 4D). For example, the presence of caster 42103 (FIG. 4A) can imply a minimum limit, which in some configurations can be set at about 1 meter. In some configurations, points in point cube 40307 (FIG. 4D) can be limited to points 1 meter or more from camera 40227 (FIG. 4D) and 6 meters or less from camera 40227 (FIG. 4D).

Continuing with reference to Figures 4C and 4D, the travel surface processor 41029 (Figure 4C) may include a base link transform 40219 that may transform the 3D cube of points into coordinates associated with the device 42114 (Figure 4A), i.e., the base link frame 40309 (Figure 4C). The base link transform 40219 may transform a 3D data point 40223 (Figure 4D) in the cube 40307 (Figure 4D) to a point 40233 (Figure 4D) in the cube 40308 (Figure 4D) whose Z dimension is set to the base of the device 42114 (Figure 4A). The travel surface processor 41029 (FIG. 4C) may include an OG preparation 40239 that may project points 40233 (FIG. 4D) in the cube 40308 (FIG. 4D) onto an occupancy grid 40244 (FIG. 4D) as points 40237 (FIG. 4D) in the cube 40311 (FIG. 4D). The layer processor 40241 may flatten the points 40237 (FIG. 4D) into various layers 40312 (FIG. 4C) depending on the data represented by the points 40237 (FIG. 4D). In some configurations, the layer processor 40241 may apply a scalar value to the layers 40312 (FIG. 4C). In some configurations, the layer 40312 (FIG. 4C) can include the probability of occupancy layer 40243, a surface classification layer 40245 (FIG. 4D) as determined by the neural network processor 40207, and a surface type confidence layer 40247 (FIG. 4D). In some configurations, the surface type confidence layer 40247 (FIG. 4D) can be determined by converting the class scores from the neural network processor 40207 to scores that can be determined by normalizing the class scores to a probability distribution over the output classes as log(class score)/Σlog(each class). In some configurations, one or more layers can be replaced or augmented by a layer that provides a probability of an undriveable surface.

Continuing with reference to FIGS. 4C and 4D, in some configurations, the probability values in occupancy layer 40243 can be expressed as log odds (log odds->ln(p/(1-p)) values. In some configurations, the probability values in occupancy layer 40243 can be based on a combination of at least mode 41033 (FIG. 4B) and surface type 40121 (FIG. 4B). In some configurations, the preselected probability values in occupancy layer 40243 can be, for example and without limitation, However, (1) when surface type 40121 (FIG. 4A) is hard and runnable and device 42114 (FIG. 4A) is in a preselected set of modes 41033 (FIG. 4B), or (2) when surface type 40121 (FIG. 4A) is soft and runnable and device 42114 (FIG. 4A) is in a specific preselected mode, such as, for example, the standard mode, or (3) when surface type 40121 (FIG. 4A) is soft and runnable and device 42114 (FIG. 4A) is in a specific preselected mode, such as, for example, the standard mode. The probability values may be selected to cover situations such as when the chair 42114 (FIG. 4A) is in a specific preselected mode, such as, for example, a four-wheel mode, or (4) when the surface type 40121 (FIG. 4A) is discontinuous and the device 42114 (FIG. 4A) is in a specific preselected mode, such as, for example, a standard mode, or (5) when the surface type 40121 (FIG. 4A) is discontinuous and the device 42114 (FIG. 4A) is in a specific preselected mode, such as, for example, a four-wheel mode, or (6) when the surface type 40121 (FIG. 4A) is non-drivable and the device 42114 (FIG. 4A) is in a preselected set of modes 41033 (FIG. 4B). In some configurations, the probability values may include, but are not limited to, those set forth in Table I. In some configurations, the neural network predicted probabilities may be adjusted, if necessary, to replace the probabilities listed in Table I.

Referring again to FIG. 4C, the occupancy grid processor 40242 can provide parameters in real time to the global occupancy grid processor 41025 that can affect the probability values of the occupancy grid 40244, such as, for example, but not limited to, surface type 40121 and occupancy grid information 41022. Configuration information 40125 (FIG. 4B), such as, for example, but not limited to, mode 41033 and surface type 40121, can be provided to the power base processor 41016 (FIG. 4B). The power base processor 41016 (FIG. 4B) can determine motor commands 40128 (FIG. 4B), which can set the configuration of the device 42114 (FIG. 4A), based at least on the configuration information 40125 (FIG. 4B).

4E and 4F, the device 42100A can be configured according to the present teachings to operate in a standard mode. In the standard mode, the casters 42103 and the second drive wheel 42101B can rest on the ground as the device 42100A navigates its path. The first drive wheel 42101A can be elevated a preselected amount 42102 (FIG. 4F) to clear the driving surface. The device 42100A can navigate normally on relatively firm level surfaces. When traveling in the standard mode, the occupancy grid 40244 (FIG. 4C) can reflect the surface type limits (see Table I) and thus enable adaptive selection of modes or configuration changes based on the surface type and current mode.

4G-4J, the device 42100B/C can be configured according to the present teachings to operate in a four-wheel mode. In one configuration in the four-wheel mode, the first drive wheel 42101A and the second drive wheel 42101B can be stationary on the ground as the device 42100A navigates its path. The casters 42103 can be retracted and clear the driving surface by a preselected amount 42104 (FIG. 4H). In another configuration in the four-wheel mode, the first drive wheel 42101A and the second drive wheel 42101B can be substantially stationary on the ground as the device 42100A navigates its path. The casters 42103 can be retracted and the chassis 42111 can be rotated (thus moving the casters 42103 further from the ground) to accommodate, for example, discontinuous surfaces. In this configuration, the casters 42103 can clear the driving surface by a preselected amount 42108 (FIG. 1J). The device 42100A/C can navigate successfully on a variety of surfaces, including soft and discontinuous surfaces. In another configuration in four-wheel mode, the second drive wheel 42101B can rest on the ground as the device 42100A, while the first drive wheel 42101A can be elevated as the device 42100C (FIG. 1J) navigates its path. The casters 42103 can be retracted and the chassis 42111 can be rotated (thus moving the casters 42103 further from the ground), for example, to accommodate discontinuous surfaces. When driving in four-wheel mode, the occupancy grid 40244 (FIG. 4C) can reflect the surface type (see Table I), thus allowing adaptive selection of modes, or configuration changes based on the surface type and current mode.

4K, a method 40150 for real-time control of a device configuration of a device, such as, for example, but not limited to, an AV, traveling a path based on at least one environmental factor and a device configuration, may include, but is not limited to, receiving sensor data 40151, determining a surface type based at least on the sensor data 40153, and determining a current mode based at least on the surface type and the current device configuration 40155. The method 40150 may include determining a next device configuration based at least on the current mode and surface type 40157, determining a movement command based at least on the next device configuration 40159, and changing the current device configuration to the next device configuration 40161 based at least on the movement command.

5A, annotated point data 10379 (FIG. 5B) can be provided to the device controller 10111 to respond to objects that appear on the AV's path. The annotated point data 10379 (FIG. 5B), which can be the basis for route information that can be used to instruct the AV 10101 (FIG. 1A) to navigate a path, can include, but is not limited to, navigable edges, mapped trajectories, such as, but not limited to, the mapped trajectories 10413/10415 (FIG. 5D), and labeled features, such as, but not limited to, the SDSF 10377 (FIG. 5C). The mapped trajectories 10413/10415 (FIG. 5C) can include a graph of edges of the route space and initial weights assigned to portions of the route space. The graph of edges can include characteristics, such as, but not limited to, directionality and capacity, and edges can be categorized according to these characteristics. The mapped trajectory 10413/10415 (FIG. 5C) may include cost modifiers associated with surfaces in the route space and driving modes associated with edges. Driving modes may include, but are not limited to, path following and SDSF climbing. Other modes may include, but are not limited to, operational modes such as autonomous, mapping, and waiting for intervention. Finally, a path may be selected based at least on the lower cost modifier. Features that are relatively far from the mapped trajectory 10413/10415 (FIG. 5C) may have higher cost modifiers and may receive less attention when forming a path. The initial weights may be adjusted while the AV 10101 (FIG. 1A) is operating, potentially causing modifications to the path. The adjusted weights may be used to adjust the edge/weight graph 10381 (FIG. 5B) and may be based at least on the current driving mode, the current surface, and the edge category.

Continuing with reference to FIG. 5A, the device controller 10111 can include a feature processor that can perform specific tasks related to incorporating the eccentricity of any feature into the path. In some configurations, the feature processor can include, but is not limited to, an SDSF processor 10118. In some configurations, the device controller 10111 can include, but is not limited to, an SDSF processor 10118, a sensor processor 10703, a mode controller 10122, and a base controller 10114, each of which are described herein. The SDSF processor 10118, the sensor processor 10703, and the mode controller 10122 can provide inputs to the base controller 10114.

Continuing with reference to FIG. 5A, the base controller 10114, based on inputs provided by at least the mode controller 10122, the SDSF processor 10118, and the sensor processor 10703, can determine information that the power base 10112 can use to cause the AV10101 (FIG. 1A) to travel on a path determined by the base controller 10114 based at least on the edge/weighted graph 10381 (FIG. 5B). In some configurations, the base controller 10114 can ensure that the AV10101 (FIG. 1A) follows a predetermined path from a start point to a destination, and can modify the predetermined path based on at least external and/or internal conditions. In some configurations, external conditions can include, but are not limited to, stop signals, SDSF, and obstacles in or near the path being traveled by the AV10101 (FIG. 1A). In some configurations, the internal conditions may include, but are not limited to, mode transitions that reflect the response the AV10101 (FIG. 1A) makes to the external conditions. The device controller 10111 may determine commands to send to the power base 10112 based on at least the external and internal conditions. The commands may include, but are not limited to, speed and direction commands that may instruct the AV10101 (FIG. 1A) to proceed in a commanded direction at a commanded speed. Other commands may include, for example, groups of commands that enable characteristic responses such as, for example, SDSF climbing. The base controller 10114 may determine the desired speed between waypoints of the path by conventional methods, including, but not limited to, for example, Interior Point Optimizer (IPOPT) large-scale nonlinear optimization (https://projects.coin-or.org/Ipopt). The base controller 10114 can determine the desired path based at least in part on conventional techniques such as, for example, but not limited to, techniques based on Dijkstra's algorithm, A ^* search algorithm, or breadth-first search algorithm. The base controller 10114 can form a box around the mapped trajectory 10413/10415 (FIG. 5C) to set the area within which obstacle detection may be performed. The height of the payload carrier, when adjustable, can be adjusted based, at least in part, on the commanded speed.

Continuing with reference to FIG. 5A, the base controller 10114 can translate speed and direction determinations into motor commands. For example, but not limited to, when encountering an SDSF, such as a curb or a slope, the base controller 10114 can instruct the power base 10112 in SDSF climb mode to raise the payload carrier 10173 (FIG. 1A), align the AV10101 (FIG. 1A) with the SDSF at approximately a 90° angle, and reduce speed to a relatively low level. When the AV10101 (FIG. 1A) climbs a substantially discontinuous surface, the base controller 10114 can instruct the power base 10112 to transition to a climb phase where speed is increased as increased torque is required to move the AV10101 (FIG. 1A) up the slope. When the AV10101 (FIG. 1A) encounters a relatively level surface, the base controller 10114 can reduce speed to stay on any flat portions of the SDSF. In the case of a downhill ramp associated with a flat portion, the base controller 10114 can allow speed to increase when the AV10101 (FIG. 1A) begins to descend a substantially discontinuous surface and when both wheels are on the downhill ramp. When an SDSF, such as, for example, but not limited to, a slope, is encountered, the slope can be identified and treated as a structure. The structural features can include, for example, a ramp of a preselected size. The ramp can include a slope of about 30° and can optionally be, but is not limited to, on both sides of the plateau. The device controller 10111 (FIG. 5A) can distinguish between obstacles and slopes by comparing the angle of the perceived feature to the expected slope ramp angle, which can be received from the sensor processor 10703 (FIG. 5A).

5B, the SDSF processor 10118 can locate navigable edges from the block of drivable surface formed by the mesh of polygons represented in the annotated point data 10379 that can be used to create a path for traversal by the AV 10101 (FIG. 1A). Within the SDSF buffer 10407 (FIG. 5C), which can form an area of a preselected size around the SDSF line 10377 (FIG. 5C), the navigable edges can be erased (see FIG. 5D) for special handling, assuming SDSF traversal. A closed line segment, such as segment 10409 (FIG. 5C), can be drawn to bisect the SDSF buffer 10407 (FIG. 5C) between the pair of previously determined SDSF points 10789 (FIG. 1N). In some configurations, for a closed line segment to be considered a candidate for SDSF traversal, the segment end 10411 (FIG. 5C) may fall on a non-obstructed portion of the drivable surface, there may be sufficient room for AV10101 (FIG. 1A) to travel along the line segment between adjacent SDSF points 10789 (FIG. 1N), and the area between the SDSF points 10789 (FIG. 1N) may be the drivable surface. The segment end 10411 (FIG. 5C) may be connected to the underlying feature to form a vertex and a drivable edge. For example, line segments 10461, 10463, 10465, and 10467 (FIG. 5C), which meet the traversal criteria, are shown as part of the feature in FIG. 5D. In contrast, line segment 10409 (FIG. 5C) did not meet the criteria because, at least, segment end 10411 (FIG. 5C) does not fall on the drivable surface. The overlapping SDSF buffers 10506 (FIG. 5C) may exhibit SDSF discontinuities that may penalize SDSF traversal of the SDSFs in the overlapped SDSF buffers 10506 (FIG. 5C). The SDSF lines 10377 (FIG. 5C) may be smoothed and the locations of the SDSF points 10789 (FIG. 1N) may be adjusted so that they fall a preselected distance apart, the preselected distance being based at least on the occupied area of the AV 10101 (FIG. 1A).

Continuing with reference to FIG. 5B, the SDSF processor 10118 can convert the annotated point data 10379 into an edge/weighted graph 10381, including morphological modifications for SDSF traversal. The SDSF processor 10118 can include a seventh processor 10601, an eighth processor 10702, a ninth processor 10603, and a tenth processor 10605. The seventh processor 10601 can convert coordinates of points in the annotated point data 10379 into a global coordinate system to achieve compatibility with GPS coordinates and generate a GPS-compatible data set 10602. The seventh processor 10601 can use conventional processes, such as, for example, but not limited to, affine matrix transformation and PostGIS transformation, to generate the GPS-compatible data set 10602. The World Geodetic System (WGS) can be used as a standard coordinate system because it takes into account the curvature of the earth. Maps can be stored in the Universal Transverse Mercator (UTM) coordinate system and can be switched to WGS when it is necessary to find where a specific address is located.

Now referring primarily to FIG. 5C, the eighth processor 10702 (FIG. 5B) may smooth the SDSF, determine the boundary of the SDSF 10377, and create a buffer 10407 around the SDSF boundary, increasing the cost modifier of the surface as it gets further from the SDSF boundary. The mapped trajectory 10413/10415 may be a special case lane with the lowest cost modifier. The lower cost modifier 10406 may generally be located near the SDSF boundary, while the higher cost modifier 10408 may generally be located relatively far from the SDSF boundary. The eighth processor 10702 may provide the point cloud data with cost 10704 (FIG. 5B) to the ninth processor 10603 (FIG. 5B).

Continuing with reference primarily to FIG. 5C, the ninth processor 10603 (FIG. 5B) can calculate an approximately 90° approach 10604 (FIG. 5B) for AV10101 (FIG. 1A) to traverse SDSF10377 that meets the criteria for labeling them as traversable. The criteria can include SDSF width and SDSF smoothness. Line segments, such as line segment 10409, can be created whose lengths indicate the minimum approach distance that AV10101 (FIG. 1A) may require to approach SDSF10377 and the minimum exit distance that may be required to exit SDSF10377. Segment end points, such as end point 10411, can be integrated with underlying routing configurations. The criteria used to determine whether an SDSF approach is possible can eliminate the possibility of some approaches. SDSF buffers, such as SDSF buffer 10407, can be used to calculate valid approach and route form edge creation.

5B, the tenth processor 10605 can create an edge/weight graph 10381 from the morphology, which is a graph of edges and weights developed herein that can be used to calculate a path through the map. The morphology can include a cost modifier and a driving mode, and the edges can include a directionality and a capacity. The weights can be adjusted at runtime based on information from any number of sources. The tenth processor 10605 can provide at least one sequence of ordered points to the base controller 10114 to enable path generation, in addition to a recommended driving mode at a particular point. Each point in each sequence of points represents a location and labeling of a possible path point on the processed drivable surface. In some configurations, the labeling can indicate that the point represents a portion of a feature that may be encountered along a path, such as, for example, but not limited to, an SDSF. In some configurations, the features can be further labeled with suggested processing based on the type of feature. For example, in some configurations, if a path point is labeled as SDSF, further labeling may include a mode. The mode may be interpreted by AV10101 (FIG. 1A) as a suggested driving instruction for AV10101 (FIG. 1A), such as, for example, switching AV10101 (FIG. 1A) to SDSF climb mode 100-31 (FIG. 5E) to enable AV10101 (FIG. 1A) to traverse SDSF 10377 (FIG. 5C).

5E, in some configurations, the mode controller 10122 can provide instructions to the base controller 10114 (FIG. 5A) to perform a mode transition. The mode controller 10122 can establish the mode in which the AV 10101 (FIG. 1A) is traveling. For example, the mode controller 10122 can provide a change of mode indication to the base controller 10114, e.g., change between a path following mode 10100-32 and an SDSF climbing mode 10100-31 when an SDSF is identified along the travel path. In some configurations, the annotated point data 10379 (FIG. 5B) can include a mode identifier at various points along the route, e.g., when the mode is changed to accommodate the route. For example, if SDSF 10377 (FIG. 5C) is labeled in the annotated point data 10379 (FIG. 5B), the device controller 10111 can determine the mode identifier associated with the route point and possibly adjust instructions to the power base 10112 (FIG. 5A) based on the desired mode. In addition to SDSF climb mode 10100-31 and path following mode 10100-32, in some configurations, the AV 10101 (FIG. 1A) can support operational modes that may include, but are not limited to, standard mode 10100-1 and enhanced (four-wheel) mode 10100-2, as described herein. The height of the payload carrier 10173 (FIG. 1A) can be adjusted to provide the necessary clearance over obstacles and along slopes.

5F, a method 11150 for navigating an AV towards a target point that crosses at least one SDSF may include, but is not limited to, receiving 11151 SDSF information related to an SDSF, a target point location, and an AV location. The SDSF information may include, but is not limited to, a set of points classified as SDSF points, and an associated probability for each point that the point is an SDSF point, respectively. The method 11150 may include drawing 11153 a closed polygon encompassing the AV location, the target point location, and drawing a path line between the target point and the AV location. The closed polygon may include a preselected width. Table I includes possible ranges for the preselected variables discussed herein. The method 11150 may include selecting 11155 two of the SDSF points located within the polygon, and drawing 11157 an SDSF line between the two points. In some configurations, the selection of the SDSF points may be random or any other method. At 11159, if there are less than a first preselected number of points within a first preselected distance of the SDSF line, and at 11161, if there are less than a second preselected number of attempts at picking SDSF points, drawing lines between them, and having less than the first preselected number of points around the SDSF line, the method 11150 may include returning to step 11155. At 11161, if there are a second preselected number of attempts at picking SDSF points, drawing lines between them, and having less than the first preselected number of points around the SDSF line, the method 11150 may include acknowledging 11163 that no SDSF lines were detected.

5G, at 11159 (FIG. 5F), if there is a first preselected number of points or more, the method 11150 can include fitting 11165 a curve to points that fall within a first preselected distance of the SDSF line. If, at 11167, the number of points within the first preselected distance of the curve exceeds the number of points within the first preselected distance of the SDSF line, and if, at 11171, the curve intersects with a path line, and if, at 11173, there are no gaps between points on the curve that exceed a second preselected distance, the method 11150 can include identifying 11175 the curve as an SDSF line. If, at 11167, the number of points within the first preselected distance of the curve does not exceed the number of points within the first preselected distance of the SDSF line, or if, at 11171, the curve does not intersect with the path line, or if, at 11173, there is a gap between points on the curve that exceeds a second preselected distance, and if, at 11177, the SDSF line does not remain stable, and if, at 11169, the curve fit has not been attempted more than a second preselected number of attempts, the method 11150 may include returning to step 11165. A stable SDSF line is the result of a subsequent iteration that results in the same or fewer points.

5H, if the curve fit has been attempted for a second preselected number of attempts at 11169 (FIG. 5G), or if the SDSF line remains stable or deteriorates at 11177 (FIG. 5G), the method 11150 may include receiving 11179 occupancy grid information. The occupancy grid may provide a probability that an obstacle is present at a point. The occupancy grid information may enhance the SDSF and route information found within a polygon that encompasses the AV route and the SDSF when the occupancy grid includes data captured and/or calculated over a common geographic area with the polygon. The method 11150 may include selecting 11181 a point from the common geographic area and its associated probability. If the probability that an obstacle is present at the selected point is higher than a preselected percentage at 11183, and if the obstacle is located between the AV and the target point at 11185, and if the obstacle is less than a third preselected distance from the SDSF line between the SDSF line and the target point at 11186, the method 11150 can include projecting the obstacle onto the SDSF line at 11187. If the probability that the location contains an obstacle is less than or equal to a preselected percentage at 11183, or if the obstacle is not located between the AV and the target point at 11185, or if the obstacle is located at a distance equal to or greater than a third preselected distance from the SDSF line between the SDSF and the target point at 11186, and if there are additional obstacles to process at 11189, the method 11150 can include resuming processing at step 11179.

5I, at 11189 (FIG. 5H), if there are no more obstacles to process, the method 11150 may include connecting the projections and finding the end points of the connected projections along the SDSF line 11191. The method 11150 may include marking 11193 a portion of the SDSF line between the projection end points as non-traversable. The method 11150 may include marking 11195 a portion of the SDSF line outside the non-traversable segment as traversable. The method 11150 may include turning 11197 the AV to within a fifth preselected amount perpendicular to the traversable segment of the SDSF line. At 11199, if the orientation error relative to a line perpendicular to the traversable segment of the SDSF line exceeds the first preselected amount, the method 11150 may include decelerating 11251 the AV by a ninth preselected amount. The method 11150 can include driving the AV forward toward the SDSF line and slowing down 11253 a second preselected amount per meter distance between the AV and the traversable SDSF line. If, at 11255, the distance of the AV from the traversable SDSF line is less than a fourth preselected distance, and if, at 11257, the heading error is greater than or equal to a third preselected amount relative to a line perpendicular to the SDSF line, the method 11150 can include slowing down 11252 the AV a ninth preselected amount.

5J, at 11257 (FIG. 5I), if the heading error is below a third preselected amount relative to a line perpendicular to the SDSF line, the method 11150 can include ignoring the updated SDSF information and running the AV at a preselected speed 11260. At 11259, if the elevation of the front portion of the AV relative to the rear portion of the AV is between a sixth preselected amount and a fifth preselected amount, the method 11150 can include running the AV forward and increasing the speed of the AV to an eighth preselected amount per degree of elevation 11261. At 11263, if the elevation of the front relative to the rear of the AV is below the sixth preselected amount, the method 11150 can include running the AV forward at a seventh preselected speed 11265. If, at 11267, the rear of the AV is greater than a fifth preselected distance from the SDSF line, the method 11150 can include acknowledging 11269 that the AV has completed traversing the SDSF. If, at 11267, the rear of the AV is less than or equal to the fifth preselected distance from the SDSF line, the method 11150 can include returning to step 11260.

5K, a system 51100 for navigating an AV towards a target point that traverses at least one SDSF can include, but is not limited to, a route line processor 11103, an SDSF detector 11109, and an SDSF controller 11127. The system 51100 can be operatively coupled to a surface processor 11601, which can process sensor information, which can include, for example, but is not limited to, images of the surroundings of the AV10101 (FIG. 5L). The surface processor 11601 can provide real-time surface feature updates, including indications of SDSF. In some configurations, a camera can provide RGB-D data, which points can be classified according to surface type. In some configurations, the system 51100 can process points classified as SDSF and their associated probabilities. The system 51100 can be operatively coupled to a system controller 11602, which can manage aspects of the operation of the AV10101 (FIG. 5L). The system controller 11602 can maintain an occupancy grid 11138, which may include information from available sources regarding the navigable area in the vicinity of the AV 10101 (FIG. 5L). The occupancy grid 11138 may include the probability that an obstacle is present. This information, in conjunction with the SDSF information, may be used to determine whether the SDSF 10377 (FIG. 5N) may be traversed by the AV 10101 (FIG. 5L) without encountering an obstacle 11681 (FIG. 5M). The system controller 11602 may determine, based on the environment and other information, a speed limit 11148 that the AV 10101 (FIG. 5N) should not exceed. The speed limit 11148 may be used as a guide to the speed set by the system 51100, or may override it. The system 51100 can be operatively coupled to a base controller 10114, which can transmit drive commands 11144 generated by the SDSF controller 11127 to the drive components of the AV10101 (FIG. 5L). The base controller 10114 can provide information to the SDSF controller 11127 about the orientation of the AV10101 (FIG. 5L) during the SDSF traversal.

Continuing with reference to FIG. 5K, the path line processor 11103 can continuously receive surface classification points 10789 in real time, which may include, but are not limited to, points classified as SDSF. The path line processor 11103 can receive the location of the target point 11139 and the AV location 11141, for example, but not limited to, as indicated by the center 11202 (FIG. 5L) of the AV 10101 (FIG. 5L). The system 51100 can include a polygon processor 11105 that draws a polygon 11147 encompassing the AV location 11141, the location of the target point 11139, and the path 11214 between the target point 11139 and the AV location 11141. The polygon 11147 can include a preselected width. In some configurations, the preselected width can include an approximate width of the AV 10101 (FIG. 5L). SDSF point 10789 that falls within polygon 11147 can be identified.

Continuing with reference to FIG. 5K, the SDSF detector 11109 can receive the surface classification points 10789, the path 11214, the polygon 11147, and the target points 11139 and can determine the most suitable SDSF line 10377 according to the criteria described herein that are available in the incoming data. The SDSF detector 11109 can include, but is not limited to, a point processor 11111 and an SDSF line processor 11113. The point processor 11111 can include selecting two of the SDSF points 10789 that are located within the polygon 11147 and drawing an SDSF 10377 line between the two points. If there are less than a first preselected number of points within a first preselected distance of the SDSF line 10377, and if there are less than a second preselected number of attempts at selecting an SDSF point 10789, drawing a line between the two points, and having less than the first preselected number of points around the SDSF line, the point processor 11111 may again include looping through a select-draw-test loop as described herein. If there are a second preselected number of attempts at selecting an SDSF point, drawing a line between them, and having less than the first preselected number of points around the SDSF line, the point processor 11111 may include acknowledging that no SDSF line was detected.

Continuing to refer to FIG. 5K, the SDSF line processor 11113 may include fitting a curve 11609-11611 (FIG. 5L) to points 10789 that fall within a first preselected distance of the SDSF line 10377 if a first preselected number or more of points 10789 are present. If the number of points 10789 within a first preselected distance of curve 11609-11611 (FIG. 5L) exceeds the number of points 10789 within a first preselected distance of SDSF line 10377, and if curve 11609-11611 (FIG. 5L) intersects path line 11214, and if there are no gaps between points 10789 on curve 11609-11611 (FIG. 5L) that exceed a second preselected distance, SDSF line processor 11113 may include identifying curve 11609-11611 (FIG. 5L) as (for example) SDSF line 10377. If the number of points 10789 within the preselected distance of the curve 11609-11611 (FIG. 5L) does not exceed the number of points 10789 within the first preselected distance of the SDSF line 10377, or if the curve 11609-11611 (FIG. 5L) does not intersect with the path line 11214, or if there is a gap between the points 10789 on the curve 11609-11611 (FIG. 5L) that exceeds a second preselected distance, and if the SDSF line 10377 does not remain stable, and if the curve fit has not been attempted more than a second preselected number of attempts, the SDSF line processor 11113 may execute the curve fit loop again.

Continuing with reference to FIG. 5K, the SDSF controller 11127 can receive the SDSF line 10377, occupancy grid 11138, AV orientation change 11142, and speed limit 11148, and can generate SDSF commands 11144 to navigate the AV 10101 (FIG. 5L) to correctly traverse the SDSF 10377 (FIG. 5N). The SDSF controller 11127 can include, but is not limited to, an obstacle processor 11115, an SDSF approach 11131, and an SDSF traversal 11133. The obstacle processor 11115 may receive the SDSF line 10377, the target point 11139, and the occupancy grid 11138 and may determine from among the obstacles identified in the occupancy grid 11138 whether any of them may obstruct the AV 10101 (FIG. 5N) as it traverses the SDSF 10377 (FIG. 5N). The obstacle processor 11115 may include, but is not limited to, an obstacle selector 11117, an obstacle tester 11119, and a traversal locator 11121. The obstacle selector 11117 may include, but is not limited to, receiving the occupancy grid 11138 as described herein. The obstacle selector 11117 may include selecting an occupancy grid point and its associated probability from a geographic area that is common to both the occupancy grid 11138 and the polygon 11147. If the probability that an obstacle is present at a selected grid point is higher than a preselected percentage, and if the obstacle is located between AV10101 (Figure 5L) and the target point 11139, and if the obstacle is less than a third preselected distance from the SDSF line 10377 between the SDSF line 10377 and the target point 11139, the obstacle tester 11119 may include projecting the obstacle onto the SDSF line 10377 and forming a projection 11621 that intersects with the SDSF line 10377. If the probability that the location contains an obstacle is below or equal to a preselected percentage, or if an obstacle is not located between AV10101 (FIG. 5L) and the target point 11139, or if an obstacle is located at a distance equal to or greater than a third preselected distance from the SDSF line 10377 between the SDSF line 10377 and the target point 11139, the obstacle tester 11119 may include resuming execution in receiving the occupancy grid 11138 if there are additional obstacles to process.

With continued reference to FIG. 5K, the traversal locator 11121 can include connecting the projection points and locating the end points 11622/11623 (FIG. 5M) of the connected projection 11621 (FIG. 5M) along the SDSF line 10377. The traversal locator 11121 can include marking the portion 11624 (FIG. 5M) of the SDSF line 10377 between the projection end points 11622/11623 (FIG. 5M) as non-traversable. The traversal locator 11121 can include marking the portion 11626 (FIG. 5M) of the SDSF line 10377 outside the non-traversable portion 11624 (FIG. 5M) as traversable.

Continuing with reference to FIG. 5K, the SDSF approach 11131 can include transmitting an SDSF command 11144 to redirect the AV 10101 (FIG. 5N) to within a fifth preselected amount perpendicular to the traversable portion 11626 (FIG. 5N) of the SDSF line 10377. If the heading error relative to the perpendicular line 11627 (FIG. 5N), which is perpendicular to the traversable section 11626 (FIG. 5N) of the SDSF line 10377, exceeds the first preselected amount, the SDSF approach 11131 can include transmitting an SDSF command 11144 to slow down the AV 10101 (FIG. 5N) by a ninth preselected amount. In some configurations, the ninth preselected amount can range from very slow to completely stopped. The SDSF approach 11131 can include transmitting an SDSF command 11144 to drive the AV10101 (FIG. 5N) forward toward the SDSF line 10377 and transmitting an SDSF command 11144 to slow down the AV10101 (FIG. 5N) by a second preselected amount per meter traveled. If the distance between the AV10101 (FIG. 5N) and the traversable SDSF line 11626 (FIG. 5N) is less than a fourth preselected distance and if the heading error is greater than or equal to a third preselected amount relative to a line perpendicular to the SDSF line 10377, the SDSF approach 11131 can include transmitting an SDSF command 11144 to slow down the AV10101 (FIG. 5N) by a ninth preselected amount.

Continuing to refer to FIG. 5K, if the heading error falls below a third preselected amount relative to a line perpendicular to SDSF line 10377, SDSF traversal 11133 may include ignoring the updated SDSF information and sending an SDSF command 11144 to run AV10101 (FIG. 5N) at a preselected rate. If the AV orientation change 11142 indicates that the climb of the leading edge 11701 (FIG. 5N) of AV10101 (FIG. 5N) relative to the trailing edge 11703 (FIG. 5N) of AV10101 (FIG. 5N) is between a sixth preselected amount and a fifth preselected amount, the SDSF traversal 11133 may include transmitting an SDSF command 11144 to cause the AV10101 (FIG. 5N) to run forward and transmitting an SDSF command 11144 to increase the speed of the AV10101 (FIG. 5N) to a preselected rate per degree of climb. If the AV orientation change 11142 indicates that the rise of the leading edge 11701 (FIG. 5N) relative to the trailing edge 11703 (FIG. 5N) of the AV 10101 (FIG. 5N) falls below a sixth preselected amount, the SDSF traversal 11133 can include sending an SDSF command 11144 to drive the AV 10101 (FIG. 5N) forward at a seventh preselected speed. If the AV location 11141 indicates that the trailing edge 11703 (FIG. 5N) is above a fifth preselected distance from the SDSF line 10377, the SDSF traversal 11133 can include acknowledging that the AV 10101 (FIG. 5N) has completed traversing the SDSF 10377. If the AV location 11141 indicates that the trailing edge 11703 (FIG. 5N) is less than or equal to a fifth preselected distance from the SDSF line 10377, the SDSF traversal 11133 may include executing the loop again beginning with ignoring the updated SDSF information.

Some example ranges for the preselected values described herein can include, but are not limited to, those outlined in Table II.

5O, to aid in real-time data collection, in some configurations, the system of the present teachings can generate locations in three-dimensional space of various surface types in response to receiving data such as, for example, but not limited to, RGD-D camera image data. The system can rotate the images 12155 and transform them from the camera coordinate system 12157 to the UTM coordinate system 12159. The system can generate polygon files from the transformed images, which can represent three-dimensional locations associated with the surface types 12161. A method 12150 for locating features 12151 from a camera image 12155 received by the AV10101 having a pose 12163 can include, but is not limited to, receiving, by the AV10101, the camera image 12155. Each camera image 12155 can include an image timestamp 12171, and each image 12155 can include image color pixels 12167 and image depth pixels 12169. The method 12150 can include receiving a pose 12163 of the AV10101, the pose 12163 having a pose timestamp 12171, and determining a selected image 12173 by identifying an image from the camera images 12155 having an image timestamp 12165 that is closest to the pose timestamp 12171. The method 12150 can include isolating image color pixels 12167 from image depth pixels 12169 in the selected image 12173 and determining an image surface classification 12161 for the selected image 12173 by providing the image color pixels 12167 to a first machine learning model 12177 and providing the image depth pixels 12169 to a second machine learning model 12179. The method 12150 can include determining perimeter points 12181 of features in the camera image 12173, the features including feature pixels 12151 within the perimeter, each of the feature pixels 12151 having the same surface classification 12161, and each of the perimeter points 12181 having a set of coordinates 12157. The method 12150 can include converting each of the sets of coordinates 12157 to UTM coordinates 12159.

Configurations of the present teachings are directed to computer systems for performing the methods discussed in the description herein, and computer-readable media containing programs for performing these methods. Raw data and results can be stored for future retrieval and processing, printed, displayed, transferred to another computer, and/or transferred to another location. Communications links can be wired or wireless, using, for example, cellular, military, and satellite communication systems. Portions of the system can run on computers with a variable number of CPUs. Other alternative computer platforms can also be used.

The present arrangement is also directed to software for performing the methods discussed herein, and computer-readable media storing software for performing these methods. The various modules described herein may be performed on the same CPU or on different computers. In accordance with the statute, the present arrangement has been described in language that is more or less specific with respect to structural and method features. However, it should be understood that the present arrangement is not limited to the specific features shown and described, since the means disclosed herein comprise preferred forms of embodying the present arrangement.

The method can be implemented, in whole or in part, electronically. Signals representing actions taken by elements of the system and other disclosed configurations can travel via at least one live communication network. Control and data information can be executed and stored electronically on at least one computer readable medium. The system can be implemented to execute on at least one computer node in at least one live communication network. General forms of the at least one computer readable medium can include, for example, but not limited to, a floppy disk, a flexible disk, a hard disk, a magnetic tape, or any other magnetic medium, a compact disk read-only memory or any other optical medium, a punch card, a paper tape, or any other physical medium with a pattern of holes, a random access memory, a programmable read-only memory, and an erasable programmable read-only memory (EPROM), a flash EPROM, or any other memory chip or cartridge, or any other medium from which a computer can read. Additionally, at least one computer-readable medium may contain graphs in any format, including, but not limited to, Graphics Interchange Format (GIF), Joint Photographic Experts Group (JPEG), Portable Network Graphics (PNG), Scalable Vector Graphics (SVG), and Tagged Image File Format (TIFF), as appropriate and subject to appropriate licensing.

Although the present teachings have been described above in terms of specific configurations, it should be understood that they are not limited to these disclosed configurations. Numerous modifications and other configurations will occur to those skilled in the art to which this pertains, and are intended and are covered by both this disclosure and the appended claims. It is intended that the scope of the present teachings should be determined by proper interpretation and construction of the appended claims and their legal equivalents, as understood by those skilled in the art relying on the disclosure in this specification and the accompanying drawings.

Claims (26)

1. An autonomous delivery vehicle, comprising:
a power base including two front wheels, two rear wheels, and an energy storage device, the power base configured to move at a commanded speed and in a commanded direction to effect transportation of at least one object, the front wheels and/or the rear wheels being powered;
a cargo platform including a plurality of short range sensors, said cargo platform being mechanically attached to said power base;
a cargo container with a volume for receiving the at least one object, the cargo container being mounted on the cargo platform ; and
a long range sensor suite comprising a LIDAR and one or more cameras, the long range sensor suite being mounted on the cargo container ;
a controller for receiving data from the long range sensor suite and the plurality of short range sensors ;
The controller:
and an executable code, the executable code comprising:
a map processor configured to access the map, the map being generated by a map processor comprising:
a first processor configured to access point cloud data from the long range sensor suite, the point cloud data representing a surface; and
a filter configured to filter the point cloud data;
a second processor configured to form a processable portion from the filtered point cloud data;
a third processor configured to merge the processable portions into at least one polygon;
a fourth processor configured to locate and label at least one substantially discontinuous surface feature (SDSF) within the at least one polygon, if present, wherein the locating and labeling forms labeled point cloud data;
a fifth processor configured to generate graphing polygons from the labeled point cloud data;
a sixth processor configured to determine a path from a start point to an end point based at least on the graphed polygon and to traverse the at least one SDSF along the path;
An autonomous delivery vehicle comprising : The autonomous delivery vehicle of claim 1, wherein the plurality of short-range sensors are selected from a camera, a stereo camera, an RGB sensor, a radar sensor, an ultrasonic sensor, and combinations thereof. The autonomous delivery vehicle of claim 1 , wherein the controller is configured to determine a geometry of a road surface based on RGB-D data received from the plurality of short-range sensors. The autonomous delivery vehicle of claim 1, wherein the plurality of short-range sensors are provided with a cooling circuit. The filter comprises:
conditionally removing points representing transient objects and points representing outliers from the point cloud data;
and replacing the removed points with a preselected height . The second processor comprises:
Segmenting the point cloud data into the processable portions;
and removing points of a preselected height from the addressable portion . The third processor,
Reducing the size of the processable portion by analyzing outliers, voxels, and normals;
Enlarging an area from the reduced size addressable portion; and
determining an initial drivable surface from the expanded region; and
segmenting and meshing the initial drivable surface;
Locating polygons within the segmented and meshed initial drivable surface; and
and establishing at least one drivable surface based at least on the polygon. The fourth processor
sorting the initial drivable surface point cloud data according to an SDSF filter, the SDSF filter including at least three categories of points;
and locating at least one SDSF point based on whether the at least three categories of points, in combination , satisfy at least one first preselected criterion. 10. The autonomous delivery vehicle of claim 8, wherein the fourth processor comprises the executable code configured to create at least one SDSF trajectory based at least on whether a plurality of the at least one SDSF point, in combination, satisfies at least one second preselected criterion. Creating a graphing polygon is
creating at least one polygon from at least one of the drivable surfaces, the at least one polygon including an outer edge;
smoothing the outer edge; and
forming a running margin based on the smoothed outer edge;
adding said at least one SDSF track to said at least one drivable surface;
and removing an inner edge from the at least one drivable surface according to at least one third preselected criterion. 11. The autonomous delivery vehicle of claim 10 , wherein smoothing the outer edge comprises a ninth processor comprising the executable code configured to trim the outer edge outwardly to form an outward edge. 12. The autonomous delivery vehicle of claim 11 , wherein forming the smoothed outer edge traveling margin comprises a tenth processor including the executable code configured to inwardly trim the outboard edge. 1. An autonomous delivery vehicle, comprising:
a power base including two front wheels, two rear wheels, and an energy storage device, the power base configured to move at a commanded speed and in a commanded direction to effect transportation of at least one object, the front wheels and/or the rear wheels being powered;
a cargo platform including a plurality of short range sensors, said cargo platform being mechanically attached to said power base;
a cargo container with a volume for receiving the at least one object, the cargo container being mounted on the cargo platform; and
a long range sensor suite comprising a LIDAR and one or more cameras, the long range sensor suite being mounted on the cargo container;
a controller for receiving data from the long range sensor suite and the plurality of short range sensors;
Equipped with
The controller:
a subsystem configured to navigate at least one substantially discontinuous surface feature (SDSF) encountered by the autonomous delivery vehicle (AV) on a surface , the surface including the at least one SDSF, the subsystem comprising:
a first processor configured to access a route configuration, the route configuration including at least one graphed polygon including filtered point cloud data, the filtered point cloud data including labeled features, and the point cloud data including a driveable margin;
a second processor configured to transform the point cloud data to a global coordinate system;
a third processor configured to determine a boundary of the at least one SDSF, the third processor creating an SDSF buffer of a preselected size around the boundary;
a fourth processor configured to determine the at least one SDSF that may be traversed based on at least one SDSF traversal criterion; and
a fifth processor configured to generate an edge/weighted graph based on at least the at least one SDSF traversal criterion, the transformed point cloud data, and the root morphology;
and a base controller configured to plan a path from a start point to an end point based at least on the edge/weight graph. The at least one SDSF crossing criterion is
a preselected width of the at least one SDSF and a preselected smoothness of the at least one SDSF;
a minimum entry and exit distance between said at least one SDSF and said AV, said minimum entry and exit distance including a drivable surface;
14. The autonomous delivery vehicle of claim 13 , further comprising: the minimum approach distance between the at least one SDSF and the AV accommodating an approximately 90 degree approach to the at least one SDSF by the AV. 1. An autonomous delivery vehicle, comprising:
a power base including two powered front wheels, two powered rear wheels, and an energy storage device, said power base configured to move at a commanded speed;
a cargo platform including a plurality of short range sensors, said cargo platform being mechanically attached to said power base;
a cargo container with a volume for receiving one or more objects to be delivered, said cargo container being mounted on said cargo platform ;
a long range sensor suite comprising a LIDAR and one or more cameras, the long range sensor suite being mounted on the cargo container ;
a controller for receiving data from the long range sensor suite and the plurality of short range sensors ;
The controller:
and an executable code, the executable code comprising:
a map processor configured to access the map, the map being generated by a map processor comprising:
a first processor configured to access point cloud data from the long range sensor suite, the point cloud data representing a surface; and
a filter configured to filter the point cloud data;
a second processor configured to form a processable portion from the filtered point cloud data;
a third processor configured to merge the processable portions into at least one polygon;
a fourth processor configured to locate and label at least one substantially discontinuous surface feature (SDSF) within the at least one polygon, if present, wherein the locating and labeling forms labeled point cloud data;
a fifth processor configured to generate graphing polygons from the labeled point cloud data;
a sixth processor configured to determine a path from a start point to an end point based at least on the graphed polygon and to traverse the at least one SDSF along the path;
An autonomous delivery vehicle comprising : The autonomous delivery vehicle of claim 15 , wherein the plurality of short-range sensors are configured to detect at least one characteristic of the surface. 16. The autonomous delivery vehicle of claim 15 , wherein the plurality of short-range sensors are selected from a camera, a stereo camera, an RGB sensor, a radar sensor, an ultrasonic sensor, and combinations thereof. The autonomous delivery vehicle of claim 15 , wherein the controller is configured to determine a geometry of a road surface based on RGB-D data received from the plurality of short-range sensors. The autonomous delivery vehicle of claim 1 , wherein the plurality of short-range sensors are configured to detect at least one characteristic of the surface. The autonomous delivery vehicle of claim 1, further comprising a second set of powered wheels that can engage the ground while the front wheels are lifted off the ground. The first light,
a first liquid-cooled heat sink;
the plurality of short-range sensors comprises a first camera;
The autonomous delivery vehicle of claim 1 , wherein the first liquid cooled heat sink cools the first light and the first camera. 22. The autonomous delivery vehicle of claim 21 , further comprising a thermoelectric cooler between the first camera and the first liquid cooled heat sink. 22. The autonomous delivery vehicle of claim 21 , wherein the first light and the first camera are recessed within a cover with an opening configured to deflect illumination from the first light away from the first camera. 22. The autonomous delivery vehicle of claim 21 , wherein the lights are angled downward a certain amount and recessed within the cover a second amount to minimize distracting illumination to pedestrians. and at least one corner camera assembly mounted on at least one corner of a front surface of the cargo platform, the at least one corner camera assembly comprising:
An ultrasonic sensor;
Corner camera and
A second light,
22. The autonomous delivery vehicle of claim 21 , comprising: a second liquid cooled heat sink, the second liquid cooled heat sink cooling the second light and the corner camera. The autonomous delivery vehicle of claim 1, wherein the controller is configured to determine the commanded speed and the commanded direction based on at least the data, and to provide the commanded speed and the commanded direction to the power base to complete the transport.

Images