JP7832283B2 - System and method for real-time control of autonomous devices - Google Patents

Description

(Cross-reference of related applications)
This utility patent application is a continuation-in-part application of U.S. Patent Application No. 16/800,497 (Patent Attorney No. AA164), filed on 25 February 2020, titled "System and Method for Surface Feature Detection and Traversal," which is incorporated herein by reference in its entirety. This patent application is a U.S. Provisional Patent Application No. 62/872,396 (Patent Attorney No. AA028), filed on July 10, 2019, titled "Apparatus for Long and Short Range Sensors on an Autonomous Delivery Device," a U.S. Provisional Patent Application No. 62/990,485 (Patent Attorney No. AA037), filed on March 17, 2020, titled "System and Method for Managing an Occupancy Grid," and a U.S. Provisional Patent Application No. 62/990,485 (Patent Attorney No. AA037), filed on July 10, 2019, titled "System and Method for Real-Time We assert the rights of U.S. Provisional Patent Application No. 62/872,320 (Patent Attorney Registration No. Z96), titled "Control of the Configuration of an Autonomous Device."

This application is incorporated herein by reference in its entirety by the following: U.S. Patent Application No. 16/035,205 (Patent Attorney No. X80), filed on 13 July 2018 and titled “MOBILITY DEVICE”; U.S. Patent Application No. 15/787,613 (Patent Attorney No. W10), filed on 18 October 2017 and titled “MOBILITY DEVICE”; U.S. Patent Application No. 15/600,703 (Patent Attorney No. U22), filed on 20 May 2017 and titled “MOBILITY DEVICE”; and U.S. Patent Application No. 15/600,703 (Patent Attorney No. U22), filed on 17 May 2018, titled “SYSTEM AND METHOD FOR SECURE REMOTE CONTROL OF A This relates to U.S. Patent Application No. 15/982,737 (Patent Attorney No. X55), titled "MEDICAL DEVICE," U.S. Provisional Application No. 62/532,993 (Patent Attorney No. U30), filed on July 15, 2017, titled "MOBILITY DEVICE IMPROVEMENTS," U.S. Provisional Application No. 62/559,263 (Patent Attorney No. V85), filed on September 15, 2017, titled "MOBILITY DEVICE SEAT," and U.S. Provisional Application No. 62/581,670 (Patent Attorney No. W07), filed on November 4, 2017, also titled "MOBILITY DEVICE SEAT."

This instruction generally relates to AV (Aviation Vehicle) systems, and more specifically, to autonomous route planning, global occupancy grid management, on-vehicle sensors, surface feature detection and traverse, and real-time vehicle configuration changes.

(background)
Navigation for AVs and semi-autonomous vehicles (AVs) typically relies on long-range sensors, including, but not limited to, LiDAR, cameras, stereo cameras, and radar. Long-range sensors can sense objects from 4 to 100 meters away from the AV. In contrast, object avoidance and/or surface detection typically relies on short-range sensors, including, but not limited to, stereo cameras, short-range radar, and ultrasonic sensors. These short-range sensors typically observe an area or volume within approximately 5 meters of the AV. Sensors can, for example, enable the AV to orient itself within its environment and navigate roads, sidewalks, obstacles, and open spaces to reach a desired destination. Sensors can also enable the visualization of people, signs, traffic signals, obstacles, and surface features.

Surface feature cross-sections can be challenging because surface features, such as substantial discontinuous surface features (SDSFs), may be found within heterogeneous morphologies, and these morphologies may be unique to specific geographical locations. However, SDSFs, such as, for example, slopes, edges, curbs, steps, and curb-like geometric shapes (referred to herein, in an unspecified manner, as SDSFs or simply surface features), may include several typical characteristics that can aid in their identification. Surface/road conditions and surface types can be recognized and classified, for example, by fusing multisensory data, which can be complex and costly. Surface features and conditions can be used to control the physical reconstruction of AVs in real time.

Sensors can be used to enable the creation of an occupied grid that can represent the world for the purpose of path planning related to AV (Aviation, Vehicle, and Space). Path planning requires a grid that identifies space as free, occupied, or unknown. However, the probability that space is occupied can improve decision-making regarding space. A log-odds representation of probabilities can be used to increase accuracy at the numerical boundaries of probabilities 0 and 1. The probability that a cell is occupied may depend at least on new sensor information, previous sensor information, and prior occupation information.

What is required is a system that combines accumulated sensor data and real-time sensor data with changes in the vehicle's physical configuration to perform variable terrain crossing. What is required is sensor placement advantageous for achieving physical configuration changes, variable terrain crossing, and object avoidance. What is required is the ability to locate SDSFs based on a multipart model associated with several criteria for SDSF identification. What is required is the determination of candidate surface feature crossings based on criteria such as candidate crossing approach angles, candidate crossing travel surfaces on both sides of candidate surface features, and real-time determination of obstacles along the candidate crossing path. What is required is a system and method for incorporating drivable surface and device mode information into occupied grid determination.

(summary)
The AV in this instruction can autonomously navigate to a desired location. In some configurations, the AV may include a device controller, a power base, four powered wheels, two caster wheels, and a cargo container, which may include sensors, a perception subsystem, an autonomy subsystem, and a driver subsystem. In some configurations, the perception and autonomy subsystems may receive and process sensor information (perception) and map information (perception and autonomy) and provide instructions to the driver subsystem. The map information may include surface classification and associated device modes. The movement of the AV, controlled by the driver subsystem and enabled by the power base, may be sensed by the sensor subsystem and provide a feedback loop. In some configurations, the SDSF may be precisely identified from point cloud data and stored in the map, for example, according to a process described herein. The portion of the map associated with the AV's location may be provided to the AV during navigation. The perception subsystem may maintain an occupancy grid that can inform the AV of the probability that the path to be traversed is currently occupied. In some configurations, the AV may operate in several distinctly different modes. Among other benefits, the mode can enable traversing complex terrain. A combination of maps (e.g., surface classification), sensor data (sensing features surrounding the AV), an occupation grid (the probability that an upcoming path point is occupied), and the mode (whether it is in a state where it can traverse difficult terrain) can be used to identify the direction, configuration, and speed of the AV.

Regarding map preparation, in some configurations, the method of this teaching for creating a map for navigating at least one SDSF encountered by an AV is provided, wherein the AV travels along a path across a surface, the surface includes at least one SDSF, and the path includes a start point and an end point. The method may include, but is not limited to, accessing point cloud data representing the surface, filtering the point cloud data, forming the filtered point cloud data into a processable portion, and merging the processable portion into at least one concave polygon. The method may also include locating and labeling at least one SDSF within at least one concave polygon. Locating and labeling can form labeled point cloud data. The method may also include creating a graphing polygon based on at least one concave polygon, and selecting a path from the start point to the end point based on at least the graphing polygon. When navigating, the AV can traverse at least one SDSF along the path.

Filtering point cloud data may optionally include conditionally removing points representing transient objects and outliers from the point cloud data, and replacing removed points with pre-selected heights. Forming processing portions may optionally include segmenting the point cloud data into processing portions and removing points with pre-selected heights from the processing portions. Merging processing portions may optionally include reducing the size of the processing portions by analyzing outliers, voxels, and normals, expanding the area from the reduced-size processing portions, determining the initial drivable surface from the expanded area, segmenting and meshing the initial drivable surface, locating polygons within the segmented and meshed initial drivable surface, and defining the drivable surface based on at least the polygons. Localizing and labeling at least one SDSF feature optionally includes sorting point cloud data of drivable surfaces according to an SDSF filter, wherein the SDSF filter includes points of at least three categories, and localizing at least one SDSF point based on whether the points of the categories, in combination, satisfy at least one first pre-selected criterion. The method optionally includes creating at least one SDSF trajectory based on whether at least one of several SDSF points, in combination, satisfy at least one second pre-selected criterion. Creating a graphed polygon further optionally includes creating at least one polygon from at least one drivable surface. The at least one polygon may include edges. Creating a graphed polygon may include smoothing edges, forming travel margins based on the smoothed edges, adding at least one SDSF trajectory to at least one traversable surface, and removing edges from at least one traversable surface according to at least one third pre-selected criterion. Edge smoothing may optionally include trimming edges outward. Forming travel margins on smoothed edges may optionally include trimming outward-facing edges inward.

In some configurations, the system of this teaching for creating a map for navigating at least one SDSF encountered by an AV, wherein the AV travels along a path across a surface, the surface includes at least one SDSF, and the path includes a start point and an end point; the system may include, but is not limited to, a first processor for accessing point cloud data representing the surface; a first filter for filtering the point cloud data; a second processor for forming a processable portion from the filtered point cloud data; a third processor for merging the processable portion into at least one concave polygon; and a fourth processor for locating and labeling at least one SDSF within at least one concave polygon, wherein locating and labeling is formed by a fourth processor for forming labeled point cloud data; a fifth processor for creating a graphing polygon; and a path selector for selecting a path from the start point to the end point based on at least the graphing polygon. The AV can traverse at least one SDSF along the path.

The first filter may optionally include executable code that conditionally removes points representing transient objects and outliers from point cloud data, and replaces the removed points with pre-selected heights. The segmenter may optionally include executable code that segmentes the point cloud data into processable portions, and removes points with pre-selected heights from the processable portions. The third processor may optionally include executable code that reduces the size of the processable portions by analyzing outliers, voxels, and normals, expands the area from the reduced-size processable portions, determines the initial drivable surface from the expanded area, segments and meshes the initial drivable surface, locates polygons within the segmented and meshed initial drivable surface, and defines the drivable surface based on at least the polygons. The fourth processor may optionally include executable code that sorts point cloud data of drivable surfaces according to an SDSF filter, wherein the SDSF filter includes points of at least three categories, and that locates at least one SDSF point based on whether the points of the categories, in combination, satisfy at least one first pre-selected criterion. The system may optionally include executable code that creates at least one SDSF trajectory based on whether a plurality of at least one SDSF points, in combination, satisfy at least one second pre-selected criterion.

Creating a graphed polygon may optionally include, but is not limited to, creating at least one polygon from at least one traversable surface, wherein at least one polygon includes an edge; smoothing the edge; forming a traversal margin based on the smoothed edge; adding at least one SDSF traversal to at least one traversable surface; and removing the edge from at least one traversable surface according to at least one third pre-selected criterion. Smoothing the edge may optionally include, but is not limited to, trimming the edge outwards. Forming a traversal margin on the smoothed edge may optionally include, but is not limited to, trimming the outward-facing edge inwards.

In some configurations, the method of this teaching for creating a map for navigating at least one SDSF encountered by an AV is provided, wherein the AV travels a path across a surface, the surface includes at least one SDSF, the path includes a start point and an end point, and the method may include, but is not limited to, accessing a route morphology. The route morphology may include at least one graphed polygon, which may include filtered point cloud data. The point cloud data may include labeled features and drivable margins. The method may include transforming the point cloud data into a global coordinate system, determining the boundary of at least one SDSF, creating an SDSF buffer of a pre-selected size around the boundary, determining at least one SDSF that can be traversed based on at least one SDSF crossing criterion, creating an edge/weighted graph based on at least one SDSF crossing criterion, the transformed point cloud data, and the route morphology, and selecting a path from the start point to the destination point based on the edge/weighted graph.

At least one SDSF crossing criterion may optionally include a pre-selected width and a pre-selected smoothness of at least one SDSF, a minimum entry and exit distance between at least one SDSF and an AV (Aviation Vehicle), including a drivable surface, and a minimum entry distance between at least one SDSF and an AV that allows for an approximately 90° approach to at least one SDSF by an AV.

In some configurations, the system of this teaching is for creating a map for navigating at least one SDSF encountered by an AV, wherein the AV travels a path across a surface, the surface comprises at least one SDSF, the path comprises a start point and an end point, and the system may include a sixth processor for accessing a route morphology, which may include at least one graphed polygon containing filtered point cloud data, the point cloud data may include labeled features and drivable margins. The system may also include a seventh processor for transforming the point cloud data into a global coordinate system, and an eighth processor for determining the boundary of at least one SDSF. The eighth processor may create an SDSF buffer of a pre-selected size around the boundary. This system may include at least a ninth processor that determines at least one SDSF that can be traversed based on at least one SDSF traversal criterion, a tenth processor that creates an edge/weighted graph based on at least one SDSF traversal criterion, converted point cloud data, and route morphology, and a base controller that selects a path from a starting point to a destination point based on at least the edge/weighted graph.

In some configurations, the Method of this Teaching is for creating a map for navigating at least one SDSF encountered by an AV, wherein the AV travels a path across a surface, the surface comprises at least one SDSF, the path comprises a start point and an end point, and the Method may include, but is not limited to, accessing point cloud data representing the surface. The Method may include filtering the point cloud data, forming the filtered point cloud data into a tadible portion, and merging the tadible portion into at least one concave polygon. The Method may include locating and labeling at least one SDSF within at least one concave polygon. Locating and labeling can form labeled point cloud data. The Method may include creating a graphing polygon based on at least one concave polygon. The graphing polygon may form a route shape, and the point cloud data may include labeled features and traversable margins. This method may include transforming point cloud data into a global coordinate system, determining the boundary of at least one SDSF, creating an SDSF buffer of a pre-selected size around the boundary, determining at least one SDSF that can be traversed based on at least one SDSF crossing criterion, creating an edge/weighted graph based on at least one SDSF crossing criterion, the transformed point cloud data, and the route configuration, and selecting a path from the starting point to the destination point based on the edge/weighted graph.

Filtering point cloud data may optionally include conditionally removing points representing transient objects and outliers from the point cloud data, and replacing removed points with pre-selected heights. Forming processing portions may optionally include segmenting the point cloud data into processing portions and removing points with pre-selected heights from the processing portions. Merging processing portions may optionally include reducing the size of the processing portions by analyzing outliers, voxels, and normals, expanding the area from the reduced-size processing portions, determining the initial drivable surface from the expanded area, segmenting and meshing the initial drivable surface, locating polygons within the segmented and meshed initial drivable surface, and defining the drivable surface based on at least the polygons. Localizing and labeling at least one SDSF feature optionally includes sorting point cloud data of drivable surfaces according to an SDSF filter, wherein the SDSF filter includes points of at least three categories, and localizing at least one SDSF point based on whether the points of the categories, in combination, satisfy at least one first pre-selected criterion. The method optionally includes creating at least one SDSF trajectory based on whether at least one of several SDSF points, in combination, satisfy at least one second pre-selected criterion. Creating a graphed polygon further optionally includes creating at least one polygon from at least one drivable surface. The at least one polygon may include edges. Creating a graphed polygon may include smoothing edges, forming travel margins based on the smoothed edges, adding at least one SDSF trajectory to at least one drivable surface, and removing edges from at least one drivable surface according to at least one third pre-selected criterion. Edge smoothing may optionally include trimming edges outward. Forming travel margins on smoothed edges may optionally include trimming outward edges inward. The at least one SDSF crossing criterion may optionally include a pre-selected width of at least one SDSF and a pre-selected smoothness of at least one SDSF, minimum entry and exit distances between at least one SDSF and AV including the drivable surface, and a minimum entry distance between at least one SDSF and AV that can accommodate an approximately 90° approach by AV to at least one SDSF.

In some configurations, the system of this teaching is for creating a map for navigating at least one SDSF encountered by an AV, wherein the AV travels a path across a surface, the surface comprises at least one SDSF, and the path comprises a start point and an end point; the system includes, but is not limited to, a point cloud accessor for accessing point cloud data representing the surface; a first filter for filtering the point cloud data; a segmenter for forming a processable portion from the filtered point cloud data; a third processor for merging the processable portion into at least one concave polygon; and a fourth processor for locating and labeling at least one SDSF within at least one concave polygon, wherein locating and labeling may include a fourth processor for forming labeled point cloud data and a fifth processor for creating a graphing polygon. The route configuration may include at least one graphing polygon which may contain the filtered point cloud data. The point cloud data may include labeled features and drivable margins. This system may include a seventh processor that converts point cloud data to a global coordinate system, and an eighth processor that determines the boundary of at least one SDSF. The eighth processor can create an SDSF buffer of a pre-selected size around the boundary. The system may also include a ninth processor that determines at least one SDSF that can be traversed based on at least one SDSF traverse criterion, a tenth processor that creates an edge/weighted graph based on at least one SDSF traverse criterion, the converted point cloud data, and the route configuration, and a base controller that selects a path from the starting point to the destination point based on at least the edge/weighted graph.

The first filter may optionally include executable code that conditionally removes points representing transient objects and outliers from point cloud data, and replaces the removed points with pre-selected heights. The segmenter may optionally include executable code that segmentes the point cloud data into processable portions, and removes points with pre-selected heights from the processable portions. The third processor may optionally include executable code that reduces the size of the processable portions by analyzing outliers, voxels, and normals, expands the area from the reduced-size processable portions, determines the initial drivable surface from the expanded area, segments and meshes the initial drivable surface, locates polygons within the segmented and meshed initial drivable surface, and defines the drivable surface based on at least the polygons. The fourth processor may optionally include executable code that sorts point cloud data of drivable surfaces according to an SDSF filter, wherein the SDSF filter includes points of at least three categories, and that locates at least one SDSF point based on whether the points of the categories, in combination, satisfy at least one first pre-selected criterion. The system may optionally include executable code that creates at least one SDSF trajectory based on whether a plurality of at least one SDSF points, in combination, satisfy at least one second pre-selected criterion.

Creating a graphed polygon may optionally include, but is not limited to, creating at least one polygon from at least one traversable surface, wherein at least one polygon includes an edge; smoothing the edge; forming a traversal margin based on the smoothed edge; adding at least one SDSF traversal to at least one traversable surface; and removing the edge from at least one traversable surface according to at least one third pre-selected criterion. Smoothing the edge may optionally include, but is not limited to, trimming the edge outwards. Forming a traversal margin on the smoothed edge may optionally include, but is not limited to, trimming the outward-facing edge inwards.

In some configurations, SDSFs can be identified by their dimensions. For example, a curb can include a width of approximately 0.6 to 0.7 m, though this is not limited to that. In some configurations, point cloud data can be processed to locate SDSFs, and this data can be used to prepare a route for the AV from the starting point to the destination. In some configurations, the route can be included in a map and provided to the perception subsystem. As the AV travels along the route, in some configurations, SDSF traverses can be adapted through sensor-based positioning of the AV, which is partially enabled by the perception subsystem. The perception subsystem can run on at least one processor within the AV.

The AV may include, but is not limited to, a power base including two powered front wheels, two powered rear wheels, and an energy storage device, and at least one processor. The power base may be configured to move at a commanded speed. The AV may include a cargo platform mechanically mounted to the power base and including a plurality of short-range sensors. In some configurations, the AV may include a cargo container mounted on the cargo platform and having volume for receiving one or more objects to be delivered. In some configurations, the AV may include a long-range sensor suite mounted on the cargo container and may include, but is not limited to, a LIDAR and one or more cameras. The AV may include a controller capable of receiving data from the long-range sensor suite and the short-range sensor suite.

The short-range sensor suite can optionally detect at least one characteristic of the drivable surface and may optionally include a stereo camera, an IR projector, two image sensors, an RGB sensor, and a radar sensor. The short-range sensor suite can optionally supply RGB-D data to a controller. The controller can optionally determine the geometric shape of the road surface based on the RGB-D data received from the short-range sensor suite. The short-range sensor suite can optionally detect objects within 4 meters of the AV (Auditory Vehicle), and the long-range sensor suite can optionally detect objects beyond 4 meters of the AV.

The perception subsystem can use data collected by sensors to capture the occupancy grid. The occupancy grid in this instruction is configured as a 3D grid of points surrounding the AV, and the AV can occupy its center point. In some configurations, the occupancy grid can extend 10 m to the left, right, rear, and front of the AV. The grid can include approximately the height of the AV, and as it moves, it virtually moves with the AV and can represent obstacles surrounding the AV. The grid can be transformed into a 2D shape by shrinking its vertical axis, and can be divided into polygons of approximately 5 cm x 5 cm in size, for example, but not limited to these. Obstacles appearing in the 3D space around the AV can be reduced to a 2D shape. If the 2D shape overlaps with any segment of one of the polygons, the polygon can be given a value of 100 to indicate that the space is occupied. Any unfilled polygon can be assigned a value of 0 and can be referred to as free space, where AV can move.

When an AV (Audio Vehicle) navigates, it may encounter situations where a change in the AV's configuration is required. The method of real-time control of the device configuration, in some configurations, includes a chassis, at least four wheels, a first side of the chassis operably coupled to at least one of the at least four wheels, and an opposing second side of the chassis operably coupled to at least one of the at least four wheels. The method may, but is not limited to, include receiving environmental data, determining a surface type based at least on the environmental data, determining a mode based at least on the surface type and a first configuration, determining a second configuration based at least on the mode and surface type, determining a move command based at least on the second configuration, and controlling the device configuration by using move commands to change the device from a first configuration to a second configuration.

This method optionally includes incorporating an occupied grid based on at least the surface type and mode. Environmental data optionally includes RGB-D image data and road surface morphology. The configuration optionally includes two clustered pairs of at least four wheels. A first pair of the two pairs can be positioned on the first side, and a second pair of the two pairs can be positioned on the second side. The first pair may include a first front wheel and a first rear wheel, and the second pair may include a second front wheel and a second rear wheel. Control of the configuration optionally includes coordinated power supply of the first and second pairs based on at least the environmental data. Control of the configuration optionally includes transitioning from driving at least four wheels and a pair of reversing casters to driving two wheels with a clustered first pair and a clustered second pair that are rotated to lift the first and second front wheels. A pair of casters can be operably coupled to the chassis. The device can be stationary on the first rear wheel, the second rear wheel, and the pair of casters. Configuration control may optionally include rotating a pair of clusters operably coupled to the two powered wheels on the first side and the two powered wheels on the second side, based on at least environmental data.

The system of this teaching for real-time control of the configuration of an AV (Aircraft Vehicle) may include, but is not limited to, a device processor and a power base processor. The AV may include a chassis, at least four wheels, a first side of the chassis, and an opposing second side of the chassis. The device processor may receive real-time environmental data surrounding the AV, determine the surface type based at least on the environmental data, determine the mode based at least on the surface type and a first configuration, and determine a second configuration based at least on the mode and surface type. The power base processor may enable the AV to move based at least on a second configuration, and may enable the AV to change from a first configuration to a second configuration. The device processor may optionally include taking in an occupied grid based at least on the surface type and mode.

During navigation, the AV may encounter an SDSF that may require the AV to be steered for a normal crossing. In some configurations, the teaching method for navigating the AV along a path line within a travel area toward a target point crossing at least one SDSF, wherein the AV includes a leading edge and a trailing edge, and the method may include, but is not limited to, receiving SDSF information and obstacle information relating to the travel area, detecting at least one candidate SDSF from the SDSF information, and selecting an SDSF line from at least one candidate SDSF line based on at least one selection criterion. This method may include determining at least one traversable portion of a selected SDSF line based on at least one location of at least one obstacle found in the obstacle information near the selected SDSF line; orienting the AV toward the at least one traversable portion by changing the AV's direction to travel along a line perpendicular to the traversable portion and operating it at a first speed; and continuously correcting the AV's orientation based on the relationship between the orientation and the perpendicular line. This method may also include driving the AV at a second speed by adjusting the first speed of the AV based on at least the orientation and the distance between the AV and the traversable portion. If the SDSF associated with at least one traversable portion rises relative to the surface of the travel route, this method may include traversing the SDSF by raising the leading edge relative to the trailing edge and driving the AV at a third increased speed per degree of rise; and driving the AV at a fourth speed until the AV has passed the SDSF.

Detecting at least one candidate SDSF from SDSF information may optionally include (a) drawing a closed polygon encompassing the location of the AV and the location of the target point; (b) drawing a path line between the target point and the location of the AV; (c) selecting two SDSF points from the SDSF information, wherein the SDSF points are located within the polygon; and (d) drawing an SDSF line between the two points. Detecting at least one candidate SDSF may also include repeating steps (c)–(e) if (e) there are fewer than a first pre-selected number of points within a first pre-selected distance of the SDSF line, and there are fewer than a second pre-selected number of attempts in selecting the SDSF points, drawing a line between them, and having fewer than a first pre-selected number of points around the SDSF line. Detecting at least one candidate SDSF may include (f) fitting a curve to SDSF points that fall within a first pre-selected distance of an SDSF line if there are a first pre-selected number of points or more; and (g) identifying a curve as an SDSF line if a first number of SDSF points within a first pre-selected distance of a curve exceeds a second number of SDSF points within a first pre-selected distance of an SDSF line, and the curve intersects a path line, and there are no gaps between SDSF points on the curve beyond a second pre-selected distance. Detecting at least one candidate SDSF may include repeating step (f)-(h) if (h) the number of points within a first pre-selected distance of the curve does not exceed the number of points within a first pre-selected distance of the SDSF line, or if the curve does not intersect the path line, or if there are gaps between SDSF points on the curve that exceed a second pre-selected distance, and if the SDSF line is not stable, and if step (f)-(h) has not been attempted more than a second pre-selected number of times.

A closed polygon may optionally include a pre-selected width, and the pre-selected width may optionally include the width dimension of the AV. The selection of SDSF points may optionally include random selection. At least one selection criterion may optionally include: a first number of SDSF points within a first pre-selected distance of the curve exceeds a second number of SDSF points within a first pre-selected distance of the SDSF line; the curve intersects the path line; and there are no gaps between SDSF points on the curve beyond a second pre-selected distance.

Determining at least one traversable portion of a selected SDSF optionally includes selecting multiple obstacle points from obstacle information. Each of the multiple obstacle points may include the probability that the obstacle point is associated with at least one obstacle. Determining at least one traversable portion may include projecting the multiple obstacle points onto the SDSF line to form at least one projection if the probability is higher than a pre-selected percentage, if any of the multiple obstacle points are located between the SDSF line and the target point, and if any of the multiple obstacle points are less than a third pre-selected distance from the SDSF line. Determining at least one traversable portion may optionally include connecting at least two of the at least one projection, locating the endpoints of the at least two connected projections along the SDSF line, marking the at least two connected projections as non-traversable SDSF sections, and marking the SDSF line outside the non-traversable sections as at least one traversable section.

Crossing at least one traversable portion of an SDSF may optionally include: redirecting the AV to proceed along a line perpendicular to the traversable portion, oriented the AV toward the traversable portion, operating it at a first speed; constantly correcting the AV's orientation based on the relationship between the orientation and the perpendicular line; and adjusting the first speed of the AV based on at least the orientation and the distance between the AV and the traversable portion to drive the AV at a second speed. Crossing at least one traversable portion of an SDSF may optionally include: crossing the SDSF by raising the leading edge relative to the trailing edge if the SDSF rises relative to the surface of the route, and driving the AV at a third increased speed per degree of rise; and driving the AV at a fourth speed until the AV has passed the SDSF.

Crossing at least one traversable portion of the SDSF may, optionally, include: (a) if the azimuth error is less than a third pre-selected amount relative to a line perpendicular to the SDSF line, ignoring the update of the SDSF information and driving the AV at a pre-selected speed; (b) if the rise of the front portion of the AV relative to the rear portion is between a sixth pre-selected amount and a fifth pre-selected amount, driving the AV forward and increasing the speed of the AV to an eighth pre-selected speed per degree of rise; (c) if the rise of the front portion is less than a sixth pre-selected amount relative to the rear portion, driving the AV forward at a seventh pre-selected speed; and (d) if the rear portion is less than or equal to a fifth pre-selected distance from the SDSF line, repeating steps (a)–(d).

In some configurations, the wheels of the SDSF and AV can be automatically aligned to avoid system instability. Automatic alignment can be implemented, for example, by continuously testing and correcting the orientation of the AV as it approaches the SDSF. Another aspect of the SDSF cross-section feature in this teaching is that it automatically verifies that sufficient free space exists around the SDSF before attempting the cross-section. Yet another aspect of the SDSF cross-section feature in this teaching is its ability to cross SDSFs of various geometric shapes. Geometric shapes can include, for example, square and contoured SDSFs. The orientation of the AV relative to the SDSF can determine the speed and direction of the AV's movement. The SDSF cross-section feature can adjust the speed of the AV in the vicinity of the SDSF. When the AV approaches the SDSF, the speed can be increased to assist the AV in crossing the SDSF.

1. An autonomous delivery vehicle comprising: a power base including two powered front wheels, two powered rear wheels, and an energy storage device, the power base being configured to move at a commanded speed and in a commanded direction and to carry out the transport of at least one object; and a cargo platform including a plurality of short-range sensors, the cargo platform being mechanically attached to the power base, and a cargo container having a volume for receiving at least one object, the cargo container being mounted on the cargo platform, and a long-range sensor suite comprising a cargo container, a LIDAR, and one or more cameras, the long-range sensor suite being mounted on the cargo container, and a controller for receiving data from the long-range sensor suite and the plurality of short-range sensors, the controller determining a commanded speed and a commanded direction based on at least the data, and the controller providing the commanded speed and a commanded direction to the power base to complete the transport; and an autonomous delivery vehicle. 2. 1. The autonomous delivery vehicle according to item 1, wherein data from multiple short-range sensors comprises at least one characteristic of the surface over which a power base travels. 3. The autonomous delivery vehicle according to item 1, wherein the multiple short-range sensors comprise at least one stereo camera. 4. The autonomous delivery vehicle according to item 1, wherein the multiple short-range sensors comprise at least one IR projector, at least one image sensor, and at least one RGB sensor. 5. The autonomous delivery vehicle according to item 1, wherein the multiple short-range sensors comprise at least one radar sensor. 6. The autonomous delivery vehicle according to item 1, wherein data from multiple short-range sensors comprises RGB-D data. 7. The autonomous delivery vehicle according to item 1, wherein the controller determines the geometric shape of the road surface based on RGB-D data received from the multiple short-range sensors. 8. The autonomous delivery vehicle according to item 1, wherein the multiple short-range sensors detect objects within 4 meters of the AV, and the long-range sensor suite detects objects beyond 4 meters from the autonomous delivery vehicle. 9. 10. The autonomous delivery vehicle according to item 1, comprising multiple short-range sensors and a cooling circuit. 11. The autonomous delivery vehicle according to item 1, comprising multiple short-range sensors and an ultrasonic sensor. The autonomous delivery vehicle according to item 2 comprises executable code, the controller being executable code, which accesses a map, the map being formed by a map processor, the map processor being a first processor that accesses point cloud data from a long-range sensor suite, the point cloud data representing a surface, a filter that filters the point cloud data, a second processor that forms a processable portion from the filtered point cloud data, a third processor that merges the processable portion into at least one polygon, and a fourth processor that, if present, locates and labels at least one substantial discontinuous surface feature (SDSF) within at least one polygon, the locating and labeling being the fourth processor that forms labeled point cloud data, a fifth processor that creates a graphed polygon from the labeled point cloud data, and a sixth processor that selects a path from a start point to an end point based on at least the graphed polygon, the AV being the sixth processor that traverses at least one SDSF along the path. 12. The autonomous delivery vehicle according to item 11, comprising a seventh processor that executes code including a filter that conditionally removes points representing transient objects and points representing outliers from point cloud data, and replaces the removed points having a pre-selected height. 13. The autonomous delivery vehicle according to item 11, comprising a second processor that includes executable code including segmenting point cloud data into processable portions and removing points of a pre-selected height from the processable portions. 14. The autonomous delivery vehicle according to item 11, comprising a third processor that includes executable code including reducing the size of the processable portions by analyzing outliers, voxels, and normals, expanding the area from the reduced-size processable portions, determining an initial drivable surface from the expanded area, segmenting and meshing the initial drivable surface, locating polygons within the segmented and meshed initial drivable surface, and setting at least one drivable surface based on at least the polygons. 15. An autonomous delivery vehicle according to item 14, comprising executable code, the fourth processor sorts point cloud data of an initial drivable surface according to an SDSF filter, wherein the SDSF filter includes points of at least three categories, and locates at least one SDSF point based on whether the points of at least three categories, in combination, satisfy at least one first pre-selected criterion. 16. An autonomous delivery vehicle according to item 15, comprising executable code, the fourth processor creates at least one SDSF trajectory based on whether a plurality of at least one SDSF points, in combination, satisfy at least one second pre-selected criterion. 17. An autonomous delivery vehicle according to item 14, comprising an eighth processor, comprising executable code, which comprises creating a graphed polygon, which means creating at least one polygon from at least one drivable surface, the at least one polygon including an outer edge, smoothing the outer edge, forming a travel margin based on the smoothed outer edge, adding at least one SDSF trajectory to at least one drivable surface, and removing an inner edge from at least one drivable surface according to at least one third pre-selected criterion. 18. An autonomous delivery vehicle according to item 17, comprising a ninth processor, comprising executable code, which comprises smoothing the outer edge, which means trimming the outer edge outward to form an outward-facing edge. 19. An autonomous delivery vehicle according to item 18, comprising a tenth processor, which comprises executable code, which means forming a travel margin on the smoothed outer edge, which means trimming the outward-facing edge inward. 20. The controller is a subsystem for navigating at least one substantial discontinuous surface feature (SDSF) encountered by an autonomous delivery vehicle (AV), wherein the AV travels a path across the surface, the surface comprises at least one SDSF, the path comprises a start point and an end point, and accesses a route morphology, the first processor comprising at least one graphed polygon comprising filtered point cloud data, the filtered point cloud data comprising labeled features, the point cloud data comprising drivable margins, and a second processor that transforms the point cloud data into a global coordinate system, and less The autonomous delivery vehicle according to item 1 comprises a subsystem comprising: a third processor for determining the boundary of one SDSF, the third processor for creating an SDSF buffer of a pre-selected size around the boundary; a fourth processor for determining at least one SDSF that can be traversed based on at least one SDSF traversal criterion; a fifth processor for creating an edge/weighted graph based on at least one SDSF traversal criterion, transformed point cloud data, and route morphology; and a base controller for selecting a path from a start point to an end point based on at least the edge/weighted graph. 21. The autonomous delivery vehicle according to item 20, wherein at least one SDSF crossing criterion comprises a pre-selected width of at least one SDSF and a pre-selected smoothness of at least one SDSF, a minimum entry distance and minimum exit distance between at least one SDSF and the AV, including a drivable surface, and a minimum entry distance between at least one SDSF and the AV that adapts an approximately 90° approach by the AV to at least one SDSF.

22. A method for managing a global occupancy grid for an autonomous device, wherein the global occupancy grid includes global occupancy grid cells, each global occupancy grid cell is associated with an occupancy probability, receives sensor data from a sensor associated with an autonomous device, and creates a local occupancy grid based at least on the sensor data, the local occupancy grid having local occupancy grid cells, and when the autonomous device moves from a first area to a second area, accesses historical data associated with the second area, creates a static grid based at least on the historical data, and moves the global occupancy grid to maintain the autonomous device at the center position of the global occupancy grid. A method comprising: maintaining; updating the moved global occupancy grid based on a static grid; marking at least one of the global occupancy grid cells as unoccupied if at least one of the global occupancy grid cells coincides with the location of an autonomous device; for each local occupancy grid cell, calculating the location of the local occupancy grid cell on the global occupancy grid; accessing a first occupancy probability from the global occupancy grid cell at that location; accessing a second occupancy probability from the local occupancy grid cell at that location; and calculating a new occupancy probability at that location on the global occupancy grid based on at least the first and second occupancy probabilities. 23. The method according to item 22, further comprising range-checking the new occupancy probability. 24. The method according to item 23, wherein the range-checking includes setting the new occupancy probability to 0 if the new occupancy probability is < 0, and setting the new occupancy probability to 1 if the new occupancy probability is > 1. 25. The method according to item 22, further comprising setting a global occupancy grid cell to the new occupancy probability. 26. The method described in item 23, further comprising setting the global occupancy grid cell to a new occupancy probability that has been range-checked.

27. A method for creating and managing occupied grids, comprising: converting sensor measurements into a reference frame associated with a device using a local occupied grid creation node; creating a timestamped measurement occupied grid; exposing the timestamped measurement occupied grid as a local occupied grid; creating multiple local occupied grids; and creating a static occupied grid based on surface characteristics in a repository, wherein the surface characteristics are associated with the location of the device; moving a global occupied grid associated with the location of the device, and maintaining the device and local occupied grids that are approximately aligned with the global occupied grid; and static occupancy A method comprising: adding information from a local grid to a global occupied grid; marking the area in the global occupied grid currently occupied by the device as unoccupied; determining the location of at least one cell in the global occupied grid for each at least one cell in each local occupied grid; accessing a first value at that location; determining a second value at that location based on the relationship between the first value and the cell value in at least one cell in the local occupied grid; comparing the second value to a pre-selected probability range; and setting up the global occupied grid with the new value if the probability value is within the pre-selected probability range. 28. The method of item 27, further comprising making the global occupied grid public. 29. The method of item 27, wherein surface properties comprise a surface type and surface discontinuities. 30. The method of item 27, wherein relationships comprise summing. 31. A system for creating and managing occupied grids, comprising multiple local grid creation nodes that create at least one local occupied grid, each local occupied grid being associated with the location of a device, and each local occupied grid containing at least one cell, and a global occupied grid manager that accesses the at least one local occupied grid, and which creates static occupied grids based on surface characteristics in a repository, each surface characteristic being associated with the location of the device, and moves the global occupied grid associated with the location of the device, and the device and at least one local that are approximately aligned with the global occupied grid. A system comprising a global occupancy grid manager that maintains local occupancy grids, adds information from static occupancy grids to at least one global occupancy grid, marks areas in the global occupancy grid currently occupied by this device as unoccupied, determines the location of at least one cell in the global occupancy grid for each at least one cell in each local occupancy grid, accesses a first value at that location, determines a second value at that location based on the relationship between the first value and the cell value in at least one cell in the local occupancy grid, compares the second value against a pre-selected probability range, and if the probability value falls within the pre-selected probability range, sets up a global occupancy grid with the new value.

32. A method for updating a global occupancy grid, comprising: updating the global occupancy grid using information from a static grid associated with a new location when an autonomous device moves to a new location; analyzing the surface at the new location; updating the surface if the surface is drivable, updating the surface and updating the global occupancy grid using the updated surface; and updating the global occupancy grid using values from a repository of static values, wherein the static values are associated with the new location. 33. The method according to item 32, wherein updating a surface includes accessing a local occupied grid associated with the new location, accessing the local occupied grid surface classification confidence value and local occupied grid surface classification for each cell in the local occupied grid, adding the global surface classification confidence value in the global occupied grid to the local occupied grid surface classification confidence value if the local occupied grid surface classification is identical to the global surface classification in the global occupied grid within the cell to form a sum, and using the sum to update the global occupied grid in the cell, subtracting the local occupied grid surface classification confidence value from the global surface classification confidence value in the global occupied grid to form a difference, and using the difference to update the global occupied grid, and if the difference is less than zero, updating the global occupied grid using the local occupied grid surface classification. 34. The method according to item 32, wherein updating the global occupancy grid using values from a repository of static values includes: accessing a log-odds value for each cell in the local occupancy grid, which is the local occupancy grid probability that the cell is an occupying value from the local occupancy grid; updating the log-odds value in the global occupancy grid using the local occupancy grid log-odds value for the cell; decreasing the log-odds that the cell is occupied in the local occupancy grid if a pre-selected certainty that the cell is not occupied is met, and the autonomous device is proceeding within the lane barrier, and the local occupancy grid surface classification indicates a drivable surface; increasing the log-odds in the local occupancy grid if the autonomous device expects to encounter a relatively uniform surface, and the local occupancy grid surface classification indicates a relatively non-uniform surface; and decreasing the log-odds in the local occupancy grid if the autonomous device expects to encounter a relatively uniform surface, and the local occupancy grid surface classification indicates a relatively uniform surface.

35. A method for real-time control of the configuration of a device, the device comprising a chassis, at least four wheels, a first side of the chassis operably coupled to at least one of the at least four wheels, and an opposing second side of the chassis operably coupled to at least one of the at least four wheels, wherein the method comprises creating a map based on at least prior surface features and an occupancy grid, the map being created non-real-time, the map including at least one location, the at least one location associated with at least one surface feature, and the at least one surface feature associated with at least one surface classification and at least one mode, the method comprising determining the current surface features as the device moves, updating the occupancy grid in real-time using the current surface features, and determining from the occupancy grid and the map a path the device may take to traverse at least one surface feature.

36. A method for real-time control of the configuration of a device, the device comprising a chassis, at least four wheels, a first side of the chassis operably coupled to at least one of the at least four wheels, and an opposing second side of the chassis operably coupled to at least one of the at least four wheels, the method comprising receiving environmental data, determining a surface type based at least on the environmental data, determining a mode based at least on the surface type and a first configuration, determining a second configuration based at least on the mode and surface type, determining a move command based at least on the second configuration, and controlling the configuration of the device by using the move command to change the device from a first configuration to a second configuration. 37. The method according to item 36, wherein the environmental data comprises RGB-D image data. 38. The method according to item 36, further comprising capturing an occupied grid based at least on the surface type and mode, and determining a move command based at least on the occupied grid. 39. The method according to item 38, wherein the occupied grid comprises information based on data from at least one image sensor. 40. The method of item 36, wherein the environmental data comprises the morphology of the road surface. 41. The configuration comprises two clustered pairs of at least four wheels, the first pair of the two pairs being positioned on the first side, the second pair of the two pairs being positioned on the second side, the first pair comprising a first front wheel and a first rear wheel, and the second pair comprising a second front wheel and a second rear wheel, the method of item 36. 42. The control of the configuration comprises coordinated power supply of the first pair and the second pair, based at least on environmental data, the method of item 41. The method according to item 41, wherein the configuration control involves driving at least four wheels and pairs of retractable casters, the pairs of casters being operably coupled to the chassis, and driving two wheels with clustered first pairs and clustered second pairs that are rotated to lift the first front wheels and second front wheels, and the device transitions to being stationary on the first rear wheels, second rear wheels, and pairs of casters. 44. The method according to item 41, wherein the configuration control involves rotating pairs of clusters operably coupled to two first powered wheels on a first side and two second powered wheels on a second side, based at least on environmental data. 45. The method according to item 36, wherein the device further comprises a cargo container, the cargo container mounted on the chassis, and the chassis controls the height of the cargo container. 46. The method according to item 45, wherein the height of the cargo container is based at least on environmental data.

47. A system for real-time control of the configuration of a device, the device comprising a chassis, at least four wheels, a first side of the chassis, and an opposing second side of the chassis, the device processor receiving real-time environmental data surrounding the device, the device processor determining a surface type based on at least the environmental data, the device processor determining a mode based on at least the surface type and a first configuration, the device processor determining a second configuration based on at least the mode and surface type, and a power base processor determining a move command based on at least the second configuration, the power base processor controlling the configuration of the device by using the move command, and changing the device from a first configuration to a second configuration. 48. The system according to item 47, wherein the environmental data comprises RGB-D image data. 49. The system according to item 47, wherein the device processor includes capturing an occupied grid based on at least the surface type and mode. 50. The system according to item 49, wherein the power base processor includes determining a move command based on at least the occupied grid. 51. The system according to item 49, wherein the occupied grid comprises information based on data from at least one image sensor. 52. The system according to item 47, wherein the environmental data comprises the form of the road surface. 53. The system according to item 47, wherein the configuration comprises two clustered pairs of at least four wheels, the first pair of the two pairs positioned on the first side, the second pair of the two pairs positioned on the second side, the first pair having a first front wheel and a first rear wheel, and the second pair having a second front wheel and a second rear wheel. 54. The system according to item 53, wherein the control of the configuration includes coordinated power supply of the first pair and the second pair based on at least environmental data. 55. The system according to item 53, wherein the control of the configuration involves driving at least four wheels and pairs of retractable casters, the pairs of casters being operably coupled to the chassis, and driving two wheels, including a clustered first pair and a clustered second pair, which are rotated to lift the first front wheels and the second front wheels, and the device transitions to being stationary on the first rear wheels, the second rear wheels, and the pairs of casters.

56. A method for maintaining a global occupancy grid, comprising: locating a first position of an autonomous device; updating the global occupancy grid using at least one occupancy probability value associated with the first position, when the autonomous device moves to a second position, the second position being associated with the global occupancy grid and the local occupancy grid; updating the global occupancy grid using at least one drivable surface associated with the local occupancy grid; updating the global occupancy grid using a surface confidence value associated with at least one drivable surface; and using a first Bayesian function to update the global occupancy grid using the log odds of at least one occupancy probability value. A method comprising updating the grid, adjusting the log odds based on at least a characteristic associated with a second location, updating the global occupancy grid using at least one drivable surface associated with the local occupancy grid when the autonomous device remains at a first location and the global occupancy grid and local occupancy grid are jointly installed, updating the global occupancy grid using a surface confidence associated with at least one drivable surface, updating the global occupancy grid using the log odds of at least one occupancy probability value using a second Bayesian function, and adjusting the log odds based on at least a characteristic associated with a second location. 57. The method according to item 35, wherein creating a map includes accessing point cloud data representing a surface, filtering the point cloud data, forming the filtered point cloud data into a processable portion, merging the processable portion into at least one concave polygon, and locating and labeling at least one SDSF within at least one concave polygon, wherein locating and labeling forms labeled point cloud data, creating a graphing polygon based on at least one concave polygon, and selecting a path from a start point to an end point based on at least one graphing polygon, wherein the AV traverses at least one SDSF along the path. 58. The method according to item 57, wherein filtering point cloud data includes conditionally removing points representing transient objects and points representing outliers from the point cloud data, and replacing the removed points with pre-selected heights. 59. Forming a processing area is the method according to item 57, comprising segmenting the point cloud data into processing areas and removing points of a pre-selected height from the processing areas. 60. Merging processing areas is the method according to item 57, comprising reducing the size of the processing areas by analyzing outliers, voxels, and normals, expanding the area from the reduced-size processing areas, determining the initial drivable surface from the expanded area, segmenting and meshing the initial drivable surface, locating polygons within the segmented and meshed initial drivable surface, and defining at least one drivable surface based on at least the polygons. 61. The method according to item 60, wherein locating and labeling at least one SDSF is a sorting of point cloud data of an initial drivable surface according to an SDSF filter, wherein the SDSF filter includes points of at least three categories, and locating at least one SDSF point on the basis that at least three categories of points together satisfy at least one first pre-selected criterion. 62. The method according to item 61, further comprising creating at least one SDSF trajectory on the basis that at least a plurality of at least one SDSF points together satisfy at least one second pre-selected criterion. 63. The method according to item 62, wherein creating a graphed polygon further comprises creating at least one polygon from at least one traversable surface, the at least one polygon including an outer edge, smoothing the outer edge, forming a traversal margin based on the smoothed outer edge, adding at least one SDSF orbital to at least one traversable surface, and removing an inner edge from at least one traversable surface according to at least one third pre-selected criterion. 64. The method according to item 63, wherein the smoothing of the outer edge comprises trimming the outer edge outward to form an outward-facing edge. 65. The method according to item 63, wherein forming a traversal margin of the smoothed outer edge comprises trimming the outward-facing edge inward.

66. An autonomous delivery vehicle comprising a power base including two powered front wheels, two powered rear wheels, and an energy storage device, wherein the power base is configured to move at a commanded speed, and a cargo platform including a plurality of short-range sensors, wherein the cargo platform is mechanically attached to the power base, and a cargo container having a volume for receiving one or more objects to be delivered, wherein the cargo container is mounted on the cargo platform, and a long-range sensor suite comprising a LIDAR and one or more cameras, wherein the long-range sensor suite is mounted on the cargo container, and a controller for receiving data from the long-range sensor suite and the plurality of short-range sensors. 67. The autonomous delivery vehicle according to item 66, wherein the plurality of short-range sensors detect at least one characteristic of a drivable surface. 68. The autonomous delivery vehicle according to item 66, wherein the plurality of short-range sensors are stereo cameras. 69. 70. The autonomous delivery vehicle according to item 66, comprising multiple short-range sensors, an IR projector, two image sensors, and an RGB sensor. 71. The autonomous delivery vehicle according to item 66, wherein the multiple short-range sensors are radar sensors. 72. The autonomous delivery vehicle according to item 66, wherein the short-range sensors supply RGB-D data to a controller. 73. The autonomous delivery vehicle according to item 66, wherein the controller determines the geometric shape of the road surface based on RGB-D data received from the multiple short-range sensors. 74. The autonomous delivery vehicle according to item 66, wherein the multiple short-range sensors detect objects within 4 meters of the autonomous delivery vehicle, and the long-range sensor suite detects objects beyond 4 meters from the autonomous delivery vehicle.

74. An autonomous delivery vehicle comprising a power base including at least two powered rear wheels, caster front wheels, and an energy storage device, wherein the power base is configured to move at a commanded speed, and a cargo platform including a plurality of short-range sensors, the cargo platform being mechanically attached to the power base, and a cargo container having a volume for receiving one or more objects to be delivered, the cargo container being mounted on the cargo platform, and a long-range sensor suite comprising a LIDAR and one or more cameras, the long-range sensor suite being mounted on the cargo container, and a controller for receiving data from the long-range sensor suite and the plurality of short-range sensors. 75. The autonomous delivery vehicle according to item 74, wherein the plurality of short-range sensors detect at least one characteristic of a drivable surface. 76. The autonomous delivery vehicle according to item 74, wherein the plurality of short-range sensors are stereo cameras. 77. 78. The autonomous delivery vehicle according to item 74, comprising multiple short-range sensors, an IR projector, two image sensors, and an RGB sensor. 79. The autonomous delivery vehicle according to item 74, wherein the multiple short-range sensors are radar sensors. 80. The autonomous delivery vehicle according to item 74, wherein the short-range sensors supply RGB-D data to a controller. 81. The autonomous delivery vehicle according to item 74, wherein the controller determines the geometric shape of the road surface based on the RGB-D data received from the multiple short-range sensors. 82. The autonomous delivery vehicle according to item 74, wherein the multiple short-range sensors detect objects within 4 meters of the autonomous delivery vehicle, and the long-range sensor suite detects objects beyond 4 meters from the autonomous delivery vehicle. 83. The autonomous delivery vehicle according to item 74, further comprising a second set of powered wheels, the caster wheels capable of engaging with the ground while being lifted away from the ground.

83. An autonomous delivery vehicle comprising: a power base including at least two powered rear wheels, caster front wheels, and an energy storage device, the power base being configured to move at a commanded speed; a cargo platform, the cargo platform being mechanically attached to the power base; and a short-range camera assembly mounted on the cargo platform for detecting at least one characteristic of a drivable surface, comprising a camera, a first light, and a first liquid-cooled heatsink, the first liquid-cooled heatsink cooling the first light and the camera; 84. The autonomous delivery vehicle according to item 83, wherein the short-range camera assembly further comprises a thermoelectric cooler between the camera and the liquid-cooled heatsink; 85. The autonomous delivery vehicle according to item 83, wherein the first light and camera are embedded in a cover with an opening that deflects illumination from the first light away from the camera; 86. The autonomous delivery vehicle according to item 83, wherein the lights are angled downward at least 15° and embedded at least 4 mm within the cover to minimize distracting lighting for pedestrians. 87. The autonomous delivery vehicle according to item 83, wherein the camera has a field of view and the first light comprises two LEDs with lenses for producing two beams of light that diffuse to illuminate the camera's field of view. 88. The autonomous delivery vehicle according to item 87, wherein the lights are angled about 50° apart and the lenses produce a 60° beam. 89. The autonomous delivery vehicle according to item 83, wherein the short-range camera assembly includes an ultrasonic sensor mounted above the camera. 90. The autonomous delivery vehicle according to item 83, wherein the short-range camera assembly is mounted in a central position on the front of the cargo platform. 91. The autonomous delivery vehicle according to item 83, further comprising at least one corner camera assembly mounted on at least one corner of the front of the cargo platform, the at least one corner camera assembly comprising an ultrasonic sensor, a corner camera, a second light, and a second liquid-cooled heat sink, the second liquid-cooled heat sink comprising a second liquid-cooled heat sink for cooling the second light and the corner camera. 92. The method according to item 22, wherein the historical data comprises surface data. 93. The method according to item 22, wherein the historical data comprises discontinuity data.
The present invention provides, for example, the following:
(Item 1)
An autonomous delivery vehicle,
A power base comprising two powered front wheels, two powered rear wheels, and an energy storage device, wherein the power base is configured to move at a commanded speed and in a commanded direction and to transport at least one object,
A cargo platform including multiple short-range sensors, wherein the cargo platform is mechanically attached to the power base,
A cargo container having a volume for receiving at least one object, wherein the cargo container is mounted on the cargo platform,
A long-range sensor suite comprising a LIDAR and one or more cameras, wherein the long-range sensor suite is mounted on the cargo container,
An autonomous delivery vehicle comprising: a controller for receiving data from the long-range sensor suite and the plurality of short-range sensors, the controller determining the commanded speed and the commanded direction based at least on the data, and the controller providing the commanded speed and the commanded direction to the power base to complete the delivery.
(Item 2)
The autonomous delivery vehicle according to item 1, wherein the data from the plurality of short-range sensors comprises at least one characteristic of the surface on which the power base travels.
(Item 3)
The autonomous delivery vehicle according to item 1, wherein the plurality of short-range sensors include at least one stereo camera.
(Item 4)
The autonomous delivery vehicle according to item 1, wherein the plurality of short-range sensors comprises at least one IR projector, at least one image sensor, and at least one RGB sensor.
(Item 5)
The autonomous delivery vehicle according to item 1, wherein the plurality of short-range sensors comprises at least one radar sensor.
(Item 6)
The data from the aforementioned multiple short-range sensors comprises RGB-D data, as described in item 1, for the autonomous delivery vehicle.
(Item 7)
The controller determines the geometric shape of the road surface based on RGB-D data received from the plurality of short-range sensors, as described in item 1, for the autonomous delivery vehicle.
(Item 8)
The autonomous delivery vehicle according to item 1, wherein the plurality of short-range sensors detect objects within 4 meters of the AV, and the long-range sensor suite detects objects beyond 4 meters from the autonomous delivery vehicle.
(Item 9)
The autonomous delivery vehicle according to item 1, wherein the plurality of short-range sensors are equipped with a cooling circuit.
(Item 10)
The autonomous delivery vehicle according to item 1, wherein the plurality of short-range sensors are equipped with ultrasonic sensors.
(Item 11)
The aforementioned controller,
It comprises executable code, and the executable code is
This includes accessing a map, the map being formed by a map processor, and the map processor
A first processor accesses point cloud data from the long-range sensor suite, wherein the point cloud data represents the surface,
A filter for filtering the aforementioned point cloud data,
A second processor that forms a processable portion from the filtered point cloud data,
A third processor that merges the processable portion into at least one polygon,
A fourth processor, if present, locates and labels the at least one substantial discontinuous surface feature (SDSF) within the at least one polygon, wherein the locating and labeling constitutes the formation of labeled point cloud data.
A fifth processor that creates a graphed polygon from the labeled point cloud data,
An autonomous delivery vehicle according to item 2, comprising a sixth processor for selecting a path from a start point to an end point based on at least the graphed polygon, wherein the AV traverses the at least one SDSF along the path.
(Item 12)
The aforementioned filter is
Conditionally remove points representing transient objects and points representing outliers from the aforementioned point cloud data,
An autonomous delivery vehicle according to item 11, comprising a seventh processor that executes a code including replacing the removed point having a pre-selected height.
(Item 13)
The second processor is,
The point cloud data is segmented into the processable portion,
An autonomous delivery vehicle according to item 11, comprising the executable code, which includes removing a point of a pre-selected height from the processable portion.
(Item 14)
The third processor described above is
By analyzing outliers, voxels, and normals, the size of the processable portion is reduced,
To expand the area from the reduced size processing portion,
Determining the initial drivable surface from the aforementioned enlarged region,
The initial drivable surface is segmented and meshed,
The process involves locating polygons within the segmented and meshed initial drivable surface,
An autonomous delivery vehicle according to item 11, comprising the executable code, which includes setting at least one drivable surface based on at least the polygon.
(Item 15)
The fourth processor is,
The method involves sorting the point cloud data of the initial drivable surface according to an SDSF filter, wherein the SDSF filter includes points of at least three categories.
An autonomous delivery vehicle according to item 14, comprising the executable code, which includes locating at least one SDSF point based on whether the points of at least three categories combined satisfy at least one first pre-selected criterion.
(Item 16)
The autonomous delivery vehicle according to item 15, comprising the executable code, the fourth processor, which includes at least creating at least one SDSF trajectory based on whether a plurality of the at least one SDSF points, in combination, satisfy at least one second pre-selected criterion.
(Item 17)
Creating a graphed polygon is,
Creating at least one polygon from the at least one drivable surface, wherein the at least one polygon includes an outer edge.
Smoothing the outer edge,
Based on the smoothed outer edge, a travel margin is formed,
Adding the at least one SDSF orbit to the at least one traversable surface,
An autonomous delivery vehicle according to item 14, comprising an eighth processor, which includes the executable code, which includes removing an inner edge from the at least one drivable surface according to at least one third pre-selected criterion.
(Item 18)
An autonomous delivery vehicle according to item 17, comprising a ninth processor, comprising the executable code, wherein smoothing the outer edge includes trimming the outer edge outward to form an outward-facing edge.
(Item 19)
An autonomous delivery vehicle according to item 18, comprising a tenth processor, comprising the executable code, wherein forming the smoothed outer edge travel margin includes trimming the outward edge inward.
(Item 20)
The aforementioned controller,
The autonomous delivery vehicle (AV) comprises a subsystem for navigating at least one substantial discontinuous surface feature (SDSF) encountered, wherein the AV travels a path across the surface, the surface includes the at least one SDSF, the path includes a start point and an end point, and the subsystem is
A first processor accessing a route configuration, wherein the route configuration includes at least one graphed polygon, the filtered point cloud data includes labeled features, and the point cloud data includes drivable margins, and the first processor accesses a route configuration, the route configuration includes at least one graphed polygon, the filtered point cloud data includes labeled features, and the point cloud data includes drivable margins,
A second processor that converts the aforementioned point cloud data into a global coordinate system,
A third processor for determining the boundary of at least one SDSF, wherein the third processor creates an SDSF buffer of a pre-selected size around the boundary.
A fourth processor that determines the at least one SDSF that can be crossed based on at least one SDSF crossing criterion,
A fifth processor that creates an edge/weighted graph based on at least one SDSF cross-sectional criterion, the converted point cloud data, and the root morphology,
An autonomous delivery vehicle according to item 1, comprising a base controller that selects the path from the starting point to the ending point based at least on the edge/weighting graph.
(Item 21)
The aforementioned at least one SDSF cross-sectional criterion is,
The pre-selected width of the at least one SDSF and the pre-selected smoothness of the at least one SDSF,
The minimum entry distance and minimum exit distance between the at least one SDSF and the AV, including the drivable surface,
An autonomous delivery vehicle according to item 20, comprising the minimum approach distance between the at least one SDSF and the AV, which adapts the AV to an approach of about 90° to the at least one SDSF.
(Item 22)
A method for managing a global occupancy grid for an autonomous device, wherein the global occupancy grid includes global occupancy grid cells, the global occupancy grid cells are associated with occupancy probabilities, and the method is
Receiving sensor data from a sensor associated with the autonomous device,
At least based on the sensor data, a local occupancy grid is created, wherein the local occupancy grid has local occupancy grid cells.
When the autonomous device moves from the first area to the second area,
Accessing historical data associated with the second area,
At least based on the aforementioned historical data, create a static grid,
To move the global occupied grid and maintain the autonomous device at the central position of the global occupied grid,
Based on the static grid, update the moved global occupancy grid,
If at least one of the globally occupied grid cells coincides with the location of the autonomous device, then at least one of the globally occupied grid cells is marked as unoccupied.
For each of the aforementioned locally occupied grid cells,
Calculating the position of the local occupied grid cell on the global occupied grid,
Accessing the first occupancy probability from the global occupancy grid cell at the aforementioned location,
Accessing the second occupancy probability from the local occupying grid cell at the aforementioned location,
A method comprising calculating a new occupancy probability at the location on the global occupancy grid based on at least the first occupancy probability and the second occupancy probability.
(Item 23)
The method according to item 22, further comprising performing a range check on the new occupancy probability.
(Item 24)
Checking the aforementioned range means
If the new occupancy probability is < 0, the new occupancy probability is set to 0.
The method according to item 23, which includes setting the new occupancy probability to 1 if the new occupancy probability is > 1.
(Item 25)
The method of item 22, further comprising setting the global occupancy grid cell to the new occupancy probability.
(Item 26)
The method of item 23, further comprising setting the global occupancy grid cell to the new occupancy probability that has been range-checked.
(Item 27)
A method for creating and managing an occupied grid,
The local dedicated grid creation node converts sensor measurements into a reference frame associated with the device,
Creating a timestamped measurement occupancy grid,
The aforementioned timestamped measurement occupancy grid is made public as a local occupancy grid,
Creating multiple local dedicated grids,
Creating a static occupancy grid based on surface characteristics within a repository, wherein the surface characteristics are associated with the location of the device,
To move the global occupancy grid associated with the location of the device, and to maintain the device and the local occupancy grid that are substantially aligned with the global occupancy grid,
Adding information from the statically occupied grid to the globally occupied grid,
To mark the area within the global occupied grid currently occupied by the aforementioned device as unoccupied,
For at least one cell in each local occupied grid,
Determining the location of at least one cell within the global occupancy grid,
Accessing the first value at the aforementioned location,
Based on the relationship between the first value and the cell value in at least one cell within the local occupied grid, a second value is determined at the location.
The second value is compared with a pre-selected probability range,
A method comprising setting up the global occupancy grid with the new value if the probability value is within the pre-selected probability range.
(Item 28)
The method of item 27, further comprising disclosing the global occupancy grid.
(Item 29)
The method according to item 27, wherein the surface characteristics include a surface type and surface discontinuity.
(Item 30)
The aforementioned relationship is the method described in item 27, which includes summing.
(Item 31)
A system for creating and managing occupied grids,
A plurality of local grid creation nodes that create at least one local occupied grid, wherein the at least one local occupied grid is associated with the location of a device, and the at least one local occupied grid includes at least one cell,
A global occupancy grid manager that accesses the at least one local occupancy grid,
The aforementioned global occupancy grid manager is
Creating a static occupancy grid based on surface characteristics within a repository, wherein the surface characteristics are associated with the location of the device,
To move the global occupancy grid associated with the location of the device, and to maintain the device and the at least one local occupancy grid that are substantially aligned with the global occupancy grid,
Adding information from the statically occupied grid to at least one globally occupied grid,
To mark the area within the global occupied grid currently occupied by the aforementioned device as unoccupied,
For each of the at least one cells in each local occupied grid,
Determining the location of at least one cell within the global occupied grid,
Accessing the first value at the aforementioned location,
Based on the relationship between the first value and the cell value in at least one cell within the local occupied grid, a second value is determined at the location.
The second value is compared with a pre-selected probability range,
A system that, if the probability value is within the pre-selected probability range, sets up the global occupancy grid with the new value.
(Item 32)
A method for updating the global occupancy grid,
When an autonomous device moves to a new location, it updates the global occupancy grid using information from the static grid associated with the new location.
Analyzing the surface at the aforementioned new location,
If the surface is drivable, the surface is updated, and the global occupied grid is updated using the updated surface.
A method comprising updating the global occupancy grid using values from a repository of static values, wherein the static values are associated with the new locations.
(Item 33)
Updating the aforementioned surface means
Accessing the local occupied grid associated with the aforementioned new location,
For each cell in the aforementioned local occupied grid,
Accessing the confidence value of the local occupied grid surface classification and the local occupied grid surface classification,
If the local occupied grid surface classification is the same as the global occupied grid surface classification within the cell, the confidence value of the global occupied grid surface classification is added to the confidence value of the local occupied grid surface classification to form a sum, and the global occupied grid in the cell is updated using the sum.
If the local occupied grid surface classification is not the same as the global occupied grid surface classification within the cell, the local occupied grid surface classification confidence value is subtracted from the global occupied grid surface classification confidence value to form a difference, and the global occupied grid is updated using the difference.
The method of item 32, comprising updating the global occupied grid using the local occupied grid surface classification if the difference is less than zero.
(Item 34)
Updating the global occupancy grid using the values from the static value repository is:
For each cell in the local occupying grid,
Accessing the log odds value, which is the local occupancy grid probability that the cell from the local occupancy grid is the occupancy value,
The log-odds value in the global occupancy grid is updated using the local occupancy grid log-odds value in the aforementioned cell.
If the pre-selected certainty that the cell is not occupied is met, and the autonomous device is moving within the lane barrier, and the local occupied grid surface classification indicates a drivable surface, then the log odds that the cell is occupied within the local occupied grid are reduced.
If the autonomous device anticipates encountering a relatively uniform surface, and the local occupied grid surface classification indicates a relatively non-uniform surface, then the log odds within the local occupied grid are increased.
The method of item 32, comprising reducing the log odds in the local occupied grid when the autonomous device expects to encounter a relatively uniform surface, and when the local occupied grid surface classification indicates a relatively uniform surface.
(Item 35)
A method for real-time control of the configuration of a device, the device comprising a chassis, at least four wheels, a first side of the chassis operably coupled to at least one of the at least four wheels, and an opposing second side of the chassis operably coupled to at least one of the at least four wheels, the method being
Creating a map based on at least prior surface features and occupancy grids, wherein the map is created non-real-time, the map includes at least one location, the at least one location is associated with at least one surface feature, and the at least one surface feature is associated with at least one surface classification and at least one mode.
As the device progresses, the current surface characteristics are determined,
The occupied grid is updated in real time using the current surface features,
A method comprising determining from the occupied grid and the map a path the device may take to traverse the at least one surface feature.
(Item 36)
A method for real-time control of the configuration of a device, the device comprising a chassis, at least four wheels, a first side of the chassis operably coupled to at least one of the at least four wheels, and an opposing second side of the chassis operably coupled to at least one of the at least four wheels, the method being
Receiving environmental data and,
The surface type is determined based at least on the aforementioned environmental data,
The mode is determined based at least on the surface type and the first configuration,
The second configuration is determined based at least on the mode and the surface type,
Determine the move command based at least on the second configuration described above,
A method comprising controlling the configuration of the device by using the move command, and changing the device from a first configuration to a second configuration.
(Item 37)
The environmental data comprises RGB-D image data, as described in item 36.
(Item 38)
Based at least the surface type and the mode, the occupied grid is incorporated,
The method of item 36, further comprising determining the move command based on at least the occupied grid.
(Item 39)
The method according to item 38, wherein the occupied grid comprises information based on data from at least one image sensor.
(Item 40)
The aforementioned environmental data comprises the form of a road surface, as described in item 36.
(Item 41)
The configuration according to item 36, comprising two clustered pairs of at least four wheels, wherein a first pair of the two pairs is positioned on the first side, a second pair of the two pairs is positioned on the second side, the first pair includes a first front wheel and a first rear wheel, and the second pair includes a second front wheel and a second rear wheel.
(Item 42)
The method according to item 41, wherein the control of the configuration includes coordinated power supply of the first pair and the second pair, based at least on the environmental data.
(Item 43)
The control of the configuration comprises a transition from driving the at least four wheels and pairs of reversible casters, the pairs of casters being operably coupled to the chassis, to driving two wheels, the clustered first pair and the clustered second pair, which are rotated to lift the first front wheel and the second front wheel, the device being stationary on the first rear wheel, the second rear wheel and the pairs of casters, according to item 41.
(Item 44)
The method according to item 41, wherein the control of the configuration includes rotating a pair of clusters operably coupled with two first powered wheels on the first side and two second powered wheels on the second side, based at least on the environmental data.
(Item 45)
The device further comprises a cargo container, the cargo container is mounted on the chassis, and the chassis controls the height of the cargo container, according to the method of item 36.
(Item 46)
The height of the cargo container is determined by the method described in item 45, based at least on the environmental data.
(Item 47)
A system for real-time control of the configuration of a device, wherein the device includes a chassis, at least four wheels, a first side of the chassis, and an opposing second side of the chassis, and the system is
A device processor that receives real-time environmental data surrounding the device, wherein the device processor determines a surface type based on at least the environmental data, determines a mode based on at least the surface type and a first configuration, and determines a second configuration based on at least the mode and the surface type.
A power base processor that determines a move command based on at least the second configuration, the power base processor controlling the configuration of the device by using the move command, and changing the device from a first configuration to a second configuration, comprising:
(Item 48)
The aforementioned environmental data is the system described in item 47, which includes RGB-D image data.
(Item 49)
The device processor includes taking in an occupied grid based at least on the surface type and the mode, according to item 47.
(Item 50)
The system according to item 49, wherein the power base processor includes determining the move command based at least on the occupied grid.
(Item 51)
The system according to item 49, wherein the occupied grid comprises information based on data from at least one image sensor.
(Item 52)
The aforementioned environmental data is provided for the system described in item 47, which includes the form of the road surface.
(Item 53)
The configuration comprises two clustered pairs of at least four wheels, the first pair of the two pairs being positioned on the first side, the second pair of the two pairs being positioned on the second side, the first pair having a first front wheel and a first rear wheel, and the second pair having a second front wheel and a second rear wheel, according to item 47.
(Item 54)
The control of the configuration includes the coordinated power supply of the first pair and the second pair, based at least on the environmental data, as described in item 53.
(Item 55)
Control of the configuration includes transitioning from driving the at least four wheels and pairs of reversible casters, the pairs of casters being operably coupled to the chassis, to driving two wheels, the clustered first pair and the clustered second pair, which are rotated to lift the first front wheel and the second front wheel, the device resting on the first rear wheel, the second rear wheel, and the pairs of casters, the system according to item 53.
(Item 56)
A method for maintaining the global grid,
To locate the first position of the autonomous device,
When the autonomous device moves to the second position, the second position is associated with the global occupancy grid and the local occupancy grid.
Updating the global occupancy grid using at least one occupancy probability value associated with the first position,
Updating the global occupancy grid using at least one drivable surface associated with the local occupancy grid,
Updating the global occupancy grid using the surface reliability associated with the at least one drivable surface,
The global occupancy grid is updated using the log odds of at least one occupancy probability value, using a first Bayesian function.
Adjusting the log odds based on at least the characteristics associated with the second position,
When the autonomous device remains in the first position and the global occupying grid and the local occupying grid are jointly installed,
Updating the global occupied grid using the at least one drivable surface associated with the local occupied grid,
Updating the global occupancy grid using the surface reliability associated with the at least one drivable surface,
The global occupancy grid is updated using the log odds of at least one occupancy probability value, using a second Bayesian function.
A method comprising adjusting the log odds based on at least the characteristics associated with the second position.
(Item 57)
Creating the aforementioned map means
Accessing point cloud data representing the aforementioned surface,
Filtering the aforementioned point cloud data,
The filtered point cloud data is formed into a processable portion,
Merging the aforementioned processable portion into at least one concave polygon,
The method involves locating and labeling the at least one SDSF within the at least one concave polygon, wherein the locating and labeling constitutes the formation of labeled point cloud data.
At a minimum, a graphed polygon is created based on at least one of the concave polygons,
The method of item 35, comprising selecting the path from a starting point to an ending point based on at least the graphed polygon, wherein the AV traverses at least one SDSF along the path.
(Item 58)
Filtering the aforementioned point cloud data is
Conditionally remove points representing transient objects and points representing outliers from the aforementioned point cloud data,
The method of item 57, comprising replacing the removed point having a pre-selected height.
(Item 59)
Forming the processing part is
The point cloud data is segmented into the processable portion,
The method according to item 57, comprising removing a point of a pre-selected height from the processable portion.
(Item 60)
Merging the aforementioned processable portions means
By analyzing outliers, voxels, and normals, the size of the processable portion is reduced,
To expand the area from the reduced size processing portion,
Determining the initial drivable surface from the aforementioned enlarged region,
The initial drivable surface is segmented and meshed,
The process involves locating polygons within the segmented and meshed initial drivable surface,
The method according to item 57, comprising setting at least one drivable surface based on at least the polygon.
(Item 61)
The location and labeling of the aforementioned at least one SDSF is
The method involves sorting the point cloud data of the initial drivable surface according to an SDSF filter, wherein the SDSF filter includes points of at least three categories.
The method of item 60, comprising at least locating at least one SDSF point based on whether the points of the at least three categories combined satisfy at least one first pre-selected criterion.
(Item 62)
The method according to item 61, further comprising creating at least one SDSF trajectory based on whether a plurality of the at least one SDSF points, in combination, satisfy at least one second pre-selected criterion.
(Item 63)
Creating the aforementioned graphed polygon further involves,
Creating at least one polygon from the at least one drivable surface, wherein the at least one polygon includes an outer edge.
Smoothing the outer edge,
Based on the smoothed outer edge, a travel margin is formed,
Adding the at least one SDSF orbit to the at least one traversable surface,
The method according to item 62, comprising removing an inner edge from the at least one drivable surface according to at least one third pre-selected criterion.
(Item 64)
The method according to item 63, wherein the smoothing of the outer edge includes trimming the outer edge outwards to form an outward-facing edge.
(Item 65)
The method according to item 63, wherein forming the running margin of the smoothed outer edge includes trimming the outward edge inward.
(Item 66)
An autonomous delivery vehicle,
A power base comprising two powered front wheels, two powered rear wheels, and an energy storage device, wherein the power base is configured to move at a commanded speed,
A cargo platform including multiple short-range sensors, wherein the cargo platform is mechanically attached to the power base,
A cargo container having volume for receiving one or more objects to be delivered, wherein the cargo container is mounted on the cargo platform,
A long-range sensor suite comprising a LIDAR and one or more cameras, wherein the long-range sensor suite is mounted on the cargo container,
An autonomous delivery vehicle comprising the long-range sensor suite and a controller for receiving data from the plurality of short-range sensors.
(Item 67)
The autonomous delivery vehicle according to item 66, wherein the plurality of short-range sensors detect at least one characteristic of a drivable surface.
(Item 68)
The autonomous delivery vehicle according to item 66, wherein the plurality of short-range sensors are stereo cameras.
(Item 69)
The autonomous delivery vehicle according to item 66 comprises an IR projector, two image sensors, and an RGB sensor.
(Item 70)
The aforementioned multiple short-range sensors are radar sensors, as described in item 66 for the autonomous delivery vehicle.
(Item 71)
The short-range sensor supplies RGB-D data to the controller, as described in item 66, for the autonomous delivery vehicle.
(Item 72)
The controller determines the geometric shape of the road surface based on RGB-D data received from the plurality of short-range sensors, as described in item 66, for the autonomous delivery vehicle.
(Item 73)
The autonomous delivery vehicle according to item 66, wherein the plurality of short-range sensors detect objects within 4 meters of the autonomous delivery vehicle, and the long-range sensor suite detects objects beyond 4 meters from the autonomous delivery vehicle.
(Item 74)
An autonomous delivery vehicle,
A power base comprising at least two powered rear wheels, a caster front wheel, and an energy storage device, wherein the power base is configured to move at a commanded speed,
A cargo platform including multiple short-range sensors, wherein the cargo platform is mechanically attached to the power base,
A cargo container having volume for receiving one or more objects to be delivered, wherein the cargo container is mounted on the cargo platform,
A long-range sensor suite comprising a LIDAR and one or more cameras, wherein the long-range sensor suite is mounted on the cargo container,
An autonomous delivery vehicle comprising the long-range sensor suite and a controller for receiving data from the plurality of short-range sensors.
(Item 75)
The autonomous delivery vehicle according to item 74, wherein the plurality of short-range sensors detect at least one characteristic of a drivable surface.
(Item 76)
The aforementioned multiple short-range sensors are stereo cameras, as described in item 74 for the autonomous delivery vehicle.
(Item 77)
The autonomous delivery vehicle according to item 74 comprises an IR projector, two image sensors, and an RGB sensor.
(Item 78)
The aforementioned multiple short-range sensors are radar sensors, as described in item 74 for the autonomous delivery vehicle.
(Item 79)
The short-range sensor supplies RGB-D data to the controller, as described in item 74, for the autonomous delivery vehicle.
(Item 80)
The controller determines the geometric shape of the road surface based on RGB-D data received from the plurality of short-range sensors, as described in item 74, for the autonomous delivery vehicle.
(Item 81)
The autonomous delivery vehicle according to item 74, wherein the plurality of short-range sensors detect objects within 4 meters of the autonomous delivery vehicle, and the long-range sensor suite detects objects beyond 4 meters from the autonomous delivery vehicle.
(Item 82)
The autonomous delivery vehicle according to item 74, further comprising a second set of powered wheels that can engage with the ground while the caster wheels are lifted away from the ground.
(Item 83)
An autonomous delivery vehicle,
A power base comprising at least two powered rear wheels, a caster front wheel, and an energy storage device, wherein the power base is configured to move at a commanded speed,
A cargo platform, wherein the cargo platform is mechanically attached to the power base,
A short-range camera assembly mounted on the cargo platform for detecting at least one characteristic of a drivable surface,
The aforementioned short-range camera assembly is
Camera and,
The first light and,
Equipped with a first liquid-cooled heatsink,
The first liquid-cooled heatsink cools the first light and the camera of an autonomous delivery vehicle.
(Item 84)
The autonomous delivery vehicle according to item 83, further comprising a thermoelectric cooler between the camera and the liquid-cooled heat sink of the short-range camera assembly.
(Item 85)
The autonomous delivery vehicle according to item 83, wherein the first light and the camera are embedded in a cover with an opening that deflects the illumination from the first light away from the camera.
(Item 86)
The lights are angled downward at least 15° and embedded at least 4 mm within the cover to minimize distracting lighting for pedestrians, as described in item 83 of the Autonomous Delivery Vehicle.
(Item 87)
The autonomous delivery vehicle according to item 83, wherein the camera has a field of view, and the first light comprises two LEDs with lenses for producing two beams of light that diffuse to illuminate the field of view of the camera.
(Item 88)
The lights are angled at a distance of approximately 50°, and the lenses produce a 60° beam, as described in item 87, for the autonomous delivery vehicle.
(Item 89)
The short-range camera assembly includes an ultrasonic sensor mounted above the camera, as described in item 83, for the autonomous delivery vehicle.
(Item 90)
The short-range camera assembly is mounted at a central position on the front of the cargo platform in the autonomous delivery vehicle according to item 83.
(Item 91)
The cargo platform further comprises at least one corner camera assembly mounted on at least one corner of the front of the cargo platform, the at least one corner camera assembly is
Ultrasonic sensor and,
Corner camera and
The second light,
The autonomous delivery vehicle according to item 83, comprising a second liquid-cooled heat sink, the second liquid-cooled heat sink comprising a second liquid-cooled heat sink for cooling the second light and the corner camera.
(Item 92)
The method for item 22, wherein the historical data comprises surface data.
(Item 93)
The method described in item 22, wherein the historical data includes discontinuous data.

This instruction will be easier to understand by referring to the following explanation, which is assumed to be accompanied by the diagrams.

Figure 1-1 is a schematic block diagram of the main components of the system described in this instruction.

Figure 1-2 is a schematic block diagram of the main components of the map processor in this instruction.

Figure 1-3 is a schematic block diagram of the main components of the perceptual processor in this instruction.

Figure 1-4 is a schematic block diagram of the main components of the autonomy processor described in this teaching.

Figure 1A is a schematic block diagram of the system of this instruction for preparing the route for AV.

Figure 1B is a pictorial illustration of an exemplary configuration of a device incorporating the system described in this instruction.

Figure 1C is a side view of an automated delivery vehicle showing the fields of view of several long-range and short-range sensors.

Figure 1D is a schematic block diagram of the map processor described in this instruction.

Figure 1E is a diagram illustrating the first part of the map processor flow of this instruction.

Figure 1F is an image of the segmented point cloud from this instruction.

Figure 1G is a diagram of the second part of the map processor in this instruction.

Figure 1H is an image showing the results of detecting drivable surfaces using this instruction.

Figure 1I is a diagram illustrating the flow of the SDSF detector described in this instruction.

Figure 1J is a pictorial illustration of the SDSF category in this instruction.

Figure 1K is an image of the SDSF identified by the system described in this instruction.

Figures 1L and 1M are illustrations of the polygon processing described in this instruction. Figures 1L and 1M are illustrations of the polygon processing described in this instruction.

Figure 1N shows images of polygons and SDSFs identified by the system of this instruction.

Figure 2A is an isometric view of the autonomous vehicle in this instruction.

Figure 2B is a top view of a cargo container showing the field of view of the selected long-range sensor.

Figure 2C-2F shows a diagram of a long-range sensor assembly. Figure 2C-2F shows a diagram of a long-range sensor assembly. Figure 2C-2F shows a diagram of a long-range sensor assembly. Figure 2C-2F shows a diagram of a long-range sensor assembly.

Figure 2G is a top view of a cargo container showing the field of view of the selected short-range sensor.

Figure 2H is an isometric view of the cargo platform in this instruction.

Figure 2I-2L is an isometric view of a short-range sensor. Figure 2I-2L is an isometric view of a short-range sensor. Figure 2I-2L is an isometric view of a short-range sensor. Figure 2I-2L is an isometric view of a short-range sensor.

Figure 2M-2N is an isometric view of the autonomous vehicle in this instruction. Figure 2M-2N is an isometric view of the autonomous vehicle in this instruction.

Figure 2O-2P is an isometric view of the autonomous vehicle of this instruction with the outer panels removed. Figure 2O-2P is an isometric view of the autonomous vehicle of this instruction with the outer panels removed.

Figure 2Q is an isometric view of the autonomous vehicle in this instruction, with a portion of the upper panel removed.

Figure 2R-2V shows the long-range sensor on the autonomous vehicle in this teaching. Figure 2R-2V shows the long-range sensor on the autonomous vehicle in this teaching. Figure 2R-2V shows the long-range sensor on the autonomous vehicle in this teaching. Figure 2R-2V shows the long-range sensor on the autonomous vehicle in this teaching. Figure 2R-2V shows the long-range sensor on the autonomous vehicle in this teaching.

Figure 2W-2Z shows an ultrasonic sensor. Figure 2W-2Z shows an ultrasonic sensor. Figure 2W-2Z shows an ultrasonic sensor. Figure 2W-2Z shows an ultrasonic sensor.

Figure 2AA-2BB shows the central short-range camera assembly. Figure 2AA-2BB shows the central short-range camera assembly .

Figure 3A is a schematic block diagram of one configuration of the system described in this instruction.

Figure 3B is a schematic block diagram of a different configuration of the system described in this instruction.

Figure 3C is a schematic block diagram of the system in this instruction, which can initially create a global occupancy grid.

Figure 3D is a pictorial representation of the static grid in this instruction.

Figures 3E and 3F are pictorial representations of the creation of the occupying grid in this instruction. Figures 3E and 3F are pictorial representations of the creation of the occupying grid in this instruction.

Figure 3G is a pictorial representation of the prior occupied grid for this instruction.

Figure 3H is a pictorial representation of updating the global occupancy grid in this instruction.

Figure 3I is a flowchart of the method described in this teaching for exposing a global occupied grid.

Figure 3J is a flowchart of the method of this teaching for updating the global occupancy grid.

Figure 3K-3M is a flowchart of another method of this teaching for updating the global occupancy grid. Figure 3K-3M is a flowchart of another method of this teaching for updating the global occupancy grid. Figure 3K-3M is a flowchart of another method of this teaching for updating the global occupancy grid.

Figure 4A is a perspective view of the device described in this instruction, showing how it can be installed in various modes.

Figure 4B is a schematic block diagram of the system described in this instruction.

Figure 4C is a schematic block diagram of the running surface processor component of this instruction.

Figure 4D is a schematic block/picture flowchart of the process described in this instruction.

Figures 4E and 4F are perspective and side views, respectively, of the device configuration described in this instruction in standard mode. Figures 4E and 4F are perspective and side views, respectively, of the device configuration described in this instruction in standard mode.

Figures 4G and 4H are perspective and side views, respectively, of the configuration of the device described in this instruction in four-wheel mode. Figures 4G and 4H are perspective and side views, respectively, of the configuration of the device of this instruction in four-wheel mode.

Figures 4I and 4J are perspective and side views, respectively, of the configuration of the device of this instruction in the lifting four-wheel mode. Figures 4I and 4J are perspective and side views, respectively, of the configuration of the device of this instruction in the lifting four-wheel mode.

Figure 4K is a flowchart of the method described in this instruction.

Figure 5A is a schematic block diagram of the device controller described in this instruction.

Figure 5B is a schematic block diagram of the SDSF processor described in this instruction.

Figure 5C is an image of the SDSF approach identified by the system of this instruction.

Figure 5D is an image of the route configuration created by the system described in this instruction.

Figure 5E is a schematic block diagram of the modes of instruction.

Figure 5F-5J is a flowchart of the teaching method for traversing the SDSF. Figure 5F-5J is a flowchart of the teaching method for traversing the SDSF. Figure 5F-5J is a flowchart of the teaching method for traversing the SDSF. Figure 5F-5J is a flowchart of the teaching method for traversing the SDSF. Figure 5F-5J is a flowchart of the teaching method for traversing the SDSF.

Figure 5K is a schematic block diagram of the system for traversing the SDSF in this instruction.

Figures 5L-5N are pictorial representations of the method shown in Figures 5F-5H. Figures 5L-5N are pictorial representations of the method shown in Figures 5F-5H. Figures 5L-5N are pictorial representations of the method shown in Figures 5F-5H.

Figure 5O is a pictorial representation that transforms an image into a polygon.

(Detailed explanation)
The system and method of this teaching can navigate the AV across surface features, including using onboard sensors and previously developed maps to develop an occupancy grid and using these aids to reconstruct the AV based on surface type and prior information.

Referring here to Figure 1-1, the AV system 100 may include a structure on which a sensor 10701 may be mounted, and on which a device controller 10111 may be executed. The structure may include a power base 10112 that can direct the movement of wheels which are part of the structure, and enable the movement of the AV. The device controller 10111 may run on at least one processor located on the AV, and may, but is not limited to, a processor located on the AV, and may receive data from the sensor 10701. The device controller 10111 may provide speed, direction, and configuration information to a base controller 10114, which may provide movement commands to the power base 10112. The device controller 10111 may receive map information from a map processor 10104, which may prepare a map of the area surrounding the AV. The device controller 10111 may include a sensor processor 10703 that can receive and process input from the sensor 10701, including, but is not limited to, a sensor on the AV. In some configurations, the device controller 10111 may include a perception processor 2143, an autonomy processor 2145, and a driver processor 2127. The perception processor 2143 can, for example, locate static and dynamic obstacles, determine traffic signal conditions, create occupancy grids, and classify surfaces. The autonomy processor 2145 can, for example, determine the maximum speed of the AV and the type of situation in which the AV is navigating, for example, on a road, on a sidewalk, at an intersection, and/or under remote control. The driver processor 2127 can, for example, create commands according to instructions from the autonomy processor 2145 and transmit them to the base controller 10114.

Referring here to Figure 1-2, the map processor 10104 can create a map of surface features and provide the map to the perception processor 2143 through the device controller 10111, which can update the occupied grid. The map processor 10104 may include, among many other aspects, a feature extractor 10801, a point cloud compiler 10803, a transient processor 10805, a segmenter 10807, a polygon generator 10809, an SDSF line generator 10811, and a combiner 10813. The feature extractor 10801 may include a first processor that accesses point cloud data representing the surface. The point cloud compiler 10803 may include a second processor that forms a processable portion from the filtered point cloud data. The transient processor 10805 may include a first filter that filters the point cloud data. The segmenter 10807 may include executable code that includes, but is not limited to, segmenting point cloud data into processable portions and removing points of a pre-selected height from the processable portions. The first filter may optionally include executable code that includes, but is not limited to, conditionally removing points representing transient objects and points representing outliers from the point cloud data and replacing the removed points of a pre-selected height. The polygon generator 10809 may include a third processor that merges the processable portions into at least one concave polygon. The third processor may optionally include executable code that includes, but is not limited to, reducing the size of the processable portions by analyzing outliers, voxels, and normals, expanding the area from the reduced-size processable portions, determining an initial drivable surface from the expanded area, segmenting and meshing the initial drivable surface, locating polygons within the segmented and meshed initial drivable surface, and defining the drivable surface based on at least the polygons. The SDSF line generator 10811 includes a fourth processor that localizes and labels at least one SDSF within at least one concave polygon, and localizing and labeling can form labeled point cloud data. The fourth processor may optionally include executable code that sorts point cloud data of drivable surfaces according to an SDSF filter, wherein the SDSF filter includes points of at least three categories, and localizes at least one SDSF point based on whether the points of the categories, in combination, satisfy at least one first pre-selected criterion. The combiner 10813 may include a fifth processor that creates graphing polygons. Creating a graphed polygon may optionally include, but is not limited to, creating at least one polygon from at least one traversable surface, wherein at least one polygon includes an edge; smoothing the edge; forming a traversal margin based on the smoothed edge; adding at least one SDSF trajectory to at least one traversable surface; and removing the edge from at least one traversable surface according to at least one third pre-selected criterion. Smoothing the edge may optionally include, but is not limited to, trimming the edge outwards, and the executable code may include that. Forming a traversal margin on the smoothed edge may optionally include, but is not limited to, trimming the outward-facing edge inwards, and the executable code may include that.

Referring here to Figure 1-3, a map can be provided to the AV, which may include onboard sensors, powered wheels, and a processor for receiving and using sensor and map data, and for powering the AV to traverse various types of surfaces, for example, when delivering goods. The onboard sensors may capture the occupied grid and provide data that can be used to detect dynamic obstacles. The occupied grid can also be captured by the map. The device controller 10111 may include a perception processor 2143 that can receive and process sensor and map data and update the occupied grid using that data.

Referring to Figure 1-4, the device controller 10111 may include a configuration processor 41023 that can automatically determine the configuration of the AV based on at least the AV mode and the surface features encountered. The autonomy processor 2145 may include a control processor 40325 that can determine the type of surface to be traversed and the configuration the AV needs to adopt to traverse the surface, based on at least a map (the planned route to be followed), information from the configuration processor 41023, and the AV mode. The autonomy processor 2145 can supply commands to the motor drive processor 40326 and implement the commands.

Referring to Figure 1A, the map processor 10104 can enable a device, for example, an AV or semi-autonomous device, to navigate within an environment that may include features such as an SDSF. Features within the map, along with onboard sensors, can enable the AV to navigate across various surfaces. In particular, the SDSF can be precisely identified and marked so that the AV can automatically maintain its performance during entry into and exit from the SDSF, and its speed, configuration, and direction can be controlled for safe SDSF traversal.

Continuing to refer to Figure 1A, in some configurations, the system 100 for managing the traverse of the SDSF may include an AV 10101, a core cloud infrastructure 10103, an AV service 10105, a device controller 10111, a sensor 10701, and a power base 10112. The AV 10101 can provide transportation and pick-up services from a starting point to a destination by following a dynamically determined route, for example, but not limited to, one that is modified by incoming sensor information. The AV 10101 may include, but not limited to, a device having an autonomous mode, a device that can operate fully autonomously, a device that can operate at least partially remotely, and a device that includes a combination of these features. The transportation device service 10105 can provide the device controller 10111 with drivable surface information, including features. The device controller 10111 can modify drivable surface information according to, for example, incoming sensor information and feature cross-sectional requirements, and select a route for the AV 10101 based on the modified drivable surface information. The device controller 10111 can present commands to the power base 10112, which instructs the power base 10112 to provide speed, direction, and configuration commands to the wheel motors and cluster motors, and the commands can cause the AV 10101 to follow the selected route and, accordingly, raise and lower its cargo. The transport device service 10105 can access route-related information from the core cloud infrastructure 10103, which may include, but is not limited to, storage and content distribution equipment. In some configurations, the core cloud infrastructure 10103 may include, for example, but not limited to, commercial products such as Amazon Web Services®, Google Cloud ^™ , and Oracle Cloud®.

Referring here to Figure 1B, an exemplary AV, which may include a device controller 10111 (Figure 1A) capable of receiving information from the map processor 10104 (Figure 1A) of this teaching, may include, for example, a power base assembly such as a power base, which is fully described in, for example, U.S. Patent Application No. 16/035,205, filed July 13, 2018, titled “Mobility Device” or U.S. Patent No. 6,571,892, filed August 15, 2001, titled “Control System and Method” (both of which are incorporated herein by reference as a whole). The exemplary power base assembly is described herein not to limit this teaching, but rather to highlight the features of any power base assembly that may be useful in implementing the art of this teaching. The exemplary power base assembly may optionally include a power base 10112, a wheel cluster assembly 11100, and a payload carrier height assembly 10068. The exemplary power base assembly may optionally provide electrical and mechanical power to drive the wheel 11203 and the cluster 11100, which can raise and lower the wheel 11203. The power base 10112 can control the rotation of the cluster assembly 11100 and the raising and lowering of the payload carrier height assembly 10068 to support the substantial discontinuous surface crossing of this teaching. Other such devices may also be used to adapt the SDSF detection and crossing of this teaching.

Referring again to Figure 1A, in some configurations, internal sensors in the exemplary power base can detect the orientation of the AV10101 and the rate of change of orientation, the motors can enable servo operation, and the controller can understand the information from the internal sensors and motors. Appropriate motor commands can be calculated to achieve transporter performance and implement path-following commands. Left and right wheel motors can drive the wheels on both sides of the AV10101. In some configurations, the front and rear wheels can be coupled to drive together such that the two left wheels can drive together and the two right wheels can drive together. In some configurations, turning can be accomplished by driving the left and right motors at different rates, and the cluster motors can rotate the wheelbase in the forward/rear direction. This can allow the AV10101 to remain horizontal while the front wheels are higher or lower than the rear wheels. This feature may be useful, for example, when ascending or descending an SDSF. The payload carrier 10173 can be automatically ascended and descended based on at least the lower terrain.

Continuing to refer to Figure 1A, in some configurations, the point cloud data may include route information relating to the area that AV 10101 should traverse. Possibly, point cloud data collected by a mapping device similar to or identical to AV 10101 may be time-tagged. The path along which the mapping device traverses may be referred to as a mapped trajectory. The point cloud data processing described herein may be performed when the mapping device traverses the mapped trajectory or after point cloud data collection is complete. After the point cloud data has been collected, it may undergo point cloud data processing, which may include initial filtering and point reduction, point cloud segmentation, and feature detection, as described herein. In some configurations, the core cloud infrastructure 10103 may provide long-term or short-term storage for the collected point cloud data and provide the data to AV service 10105. The AV service 10105 can select from possible point cloud datasets to find a dataset that covers the area surrounding a desired starting point and a desired destination related to AV 10101. The AV service 10105 may, but is not limited to, include a map processor 10104 capable of reducing the size of the point cloud data and determining the features represented within the point cloud data. In some configurations, the map processor 10104 can determine the location of SDSFs from the point cloud data. In some configurations, polygons can be created from the point cloud data as a technique for segmenting the point cloud data and ultimately defining drivable surfaces. In some configurations, SDSF detection and drivable surface determination can proceed in parallel. In some configurations, SDSF detection and drivable surface determination can proceed sequentially.

Referring here to Figure 1C, in some configurations, the AV may be configured to perform other functions involving the delivery of cargo and/or autonomous navigation to a desired location. In some applications, the AV may be remotely guided. In some configurations, the AV 20100 includes a cargo container that can be remotely, automatically, or manually released in response to user input, allowing the user to place or remove cargo and other items. The cargo container 20110 is mounted on a cargo platform 20160, which is mechanically connected to a power base 20170. The power base 20170 includes four powered wheels 20174 and two caster wheels 20176. The power base provides speed and direction control for moving the cargo container 20110 along the ground and over obstacles including curbs and other discontinuous surface features.

Continuing to refer to Figure 1C, the cargo platform 20160 is connected to the power base 20170 through two U-frames 20162. Each U-frame 20162 is rigidly attached to the structure of the cargo platform 20160 and includes two holes, allowing a rotatable joint 20164 to be formed together with the ends of each arm 20172 on the power base 20170. The power base controls the rotational position of the arms and, therefore, controls the height and tilt angle of the cargo container 20110.

Continuing to refer to Figure 1C, in some configurations, the AV20100 includes one or more processors for receiving data, navigating the path, and selecting the direction and speed of the power base 20170.

Referring to Figure 1D, in some configurations, the map processor 10104 of this teaching can position the SDSF on the map. The map processor 10104 may, but is not limited to, include a feature extractor 10801, a point cloud compiler 10803, a transient processor 10805, a segmenter 10807, a polygon generator 10809, an SDSF line generator 10811, and a data combiner 10813.

Continuing with Figure 1D, the feature extractor 10801 (Figure 1-2) may include, but is not limited to, line-of-sight filtering 10121 of the point cloud data 10131 and the mapped trajectory 10133. Line-of-sight filtering can remove points that are hidden from the direct line of sight of the sensor that collects the point cloud data and forms the mapped trajectory. The point cloud organizer 10803 (Figure 1-2) may, as possible, organize the reduced point cloud data 10132 according to pre-selected criteria associated with specific features (10151). In some configurations, the transient processor 10805 (Figure 1-2) may remove transient points from the organized point cloud data and mapped trajectory 10133 by any number of methods, including those described herein (10153). Transient points can complicate processing, in particular when the specific features are stationary. The segmenter 10807 (Figure 1-2) can divide the processed point cloud data 10135 into processable chunks. In some configurations, the processed point cloud data 10135 can be segmented into chunks having a pre-selected minimum number of points, e.g., approximately 100,000 points (10155), though not limited to these. In some configurations, further point reduction can be based on pre-selected criteria that may relate to the features to be extracted. For example, if points above a certain height are not important for localizing features, those points can be removed from the point cloud data. In some configurations, the height of at least one of the sensors collecting the point cloud data can be considered the origin, and points above the origin can be removed from the point cloud data, e.g., when only points of interest are associated with surface features. After the filtered point cloud data 10135 has been segmented, it forms segments 10137, and the remaining points can be divided into drivable surface chunks, from which surface features can be localized. In some configurations, the polygon generator 10809 (Figure 1-2) can locate traversable surfaces by generating polygons 10139 (10161), for example, as described herein (but not limited to) (10161). In some configurations, the SDSF line generator 10811 (Figure 1-2) can locate surface features by generating SDSF lines 10141 (10163), for example, as described herein (but not limited to) (10163). In some configurations, the combiner 10813 (Figure 1-2) can combine polygons 10139 and SDSF lines 10141 (10165) to create a dataset that can be further processed to generate actual paths that AV 10101 (Figure 1A) may travel.

Here, primarily referring to Figure 1E, removing transient objects with respect to the mapped trajectory 10133, such as an exemplary timestamped point 10751, from the point cloud data 10131 (Figure 1D) 10153 (Figure 1D) may include casting a ray 10753 from a timestamped point on the mapped trajectory 10133 to each timestamped point in the point cloud data 10131 (Figure 1D) that has substantially the same timestamped point. If the ray 10753 intersects a point between the timestamped point on the mapped trajectory 10133 and the endpoint of the ray 10753, for example, point D10755, then the intersection point D10755 can be assumed to have entered the point cloud data between different sweeps of the camera. The intersection point, for example, intersection point D10755, can be assumed to be part of a transient object and can be removed from the reduced point cloud data 10132 (Figure 1D) as not representing fixed features such as SDSF. The result is, for example, processed point cloud data 10135 (Figure 1D) without transient objects, though not limited to them. Points that were removed as part of transient objects but are also substantially at ground level can be returned to processed point cloud data 10135 (Figure 1D) (10754). Transient objects cannot include certain features, such as, for example, SDSF 10141 (Figure 1D), and therefore, when SDSF 10141 (Figure 1D) is a detected feature, it can be removed without interfering with the integrity of point cloud data 10131 (Figure 1D).

Continuing with Figure 1E, segmenting the processed point cloud data 10135 (Figure 1D) by 10155 (Figure 1D) can generate segments 10757 having rectangles 10154 (Figure 1F) with pre-selected size and shape, e.g., minimum pre-selected side length, and containing approximately 100,000 points. From each segment 10757, points not necessarily related to the specific task, e.g., points located above a pre-selected level, can be removed to reduce the dataset size (10157) (Figure 1D). In some configurations, the pre-selected level may be the height of AV 10101 (Figure 1A). Removing these points can lead to more efficient processing of the dataset.

Referring again, primarily to Figure 1D, the map processor 10104 can supply the device controller 10111 with at least one dataset that can be used to generate direction, speed, and configuration commands for controlling the AV 10101 (Figure 1A). At least one dataset may include points that can be connected to other points in the dataset, and each line connecting points in the dataset traverses a traversable surface. To determine such root points, the segmented point cloud data 10137 can be divided into polygons 10139, and the vertices of the polygons 10139 can potentially be root points. The polygons 10139 may include features such as, for example, an SDSF 10141.

Continuing to refer to Figure 1D, in some configurations, creating the processed point cloud data 10135 may include filtering voxels. To reduce the number of points that will undergo future processing, in some configurations, the centroid of each voxel in the dataset can be used to approximate the points within the voxel, and all points other than the centroid can be excluded from the point cloud data. In some configurations, the center of a voxel can be used to approximate the points within the voxel. Other methods for reducing the size of the filtered segment 10251 (Figure 1G) can also be used, such as taking a random point subsample so that a fixed number of points, selected uniformly and randomly (but not limited to), can be excluded from the filtered segment 10251 (Figure 1G).

Continuing to refer to Figure 1D, in some configurations, creating the processed point cloud data 10135 may involve calculating normals from a dataset from which outliers have been removed and which has been resized through voxel filtering. The normals for each point in the filtered dataset can be used for various processing possibilities, including curve reconstruction algorithms. In some configurations, estimating and filtering normals in the dataset may involve obtaining the underlying surface from the dataset using a surface meshing technique and calculating normals from the surface mesh. In some configurations, estimating normals may involve using approximations to infer surface normals directly from the dataset, such as determining the normals to a fitting plane obtained by applying the total least squares method to the k nearest neighbors of a point, for example, but not limited to. In some configurations, the value of k may be selected based on at least empirical data. Filtering normals may involve removing any normals greater than approximately 45° from those perpendicular to the x-y plane. In some configurations, the filter may be used to align normals in the same direction. If a portion of the dataset represents a planar surface, redundant information contained within adjacent normals can be filtered out either by performing random subsampling or by filtering out one point from the set of related points. In some configurations, point selection may involve recursively decomposing the dataset into boxes until each box contains at most k points. A single normal can be calculated from the k points within each box.

Continuing to refer to Figure 1D, in some configurations, creating the processed point cloud data 10135 may involve expanding the region within the dataset by clustering points that geometrically fit the surface representing the dataset, and refining the surface as the region expands to obtain the best approximation of the maximum number of points. Region expansion can merge points in terms of smoothness constraints. In some configurations, the smoothness constraints may be determined empirically, for example, or based on a desired surface smoothness. In some configurations, the smoothness constraints may range from approximately 10π/180 to approximately 20π/180. The output of region expansion is a set of point clusters, each point cluster being a set of points, each of which is considered to be part of the same smooth surface. In some configurations, region expansion may be based on a comparison of angles between normals. Region expansion can be segmented, for example, but is not limited to, http://pointclouds. This can be performed using algorithms such as those found in org/documentation/tutorials/regiment_growing_segmentation.php and http://pointclouds.org/documentation/tutorials/cluster_extraction.php#cluster-extraction.

Referring here to Figure 1G, the segmented point cloud data 10137 (Figure 1D) can be used to generate a polygon 10759, for example, a 5m × 5m polygon (10161) (Figure 1D). Point subclusters can be transformed into a polygon 10759 using, for example, meshing. Meshing can be performed by standard methods such as, but are not limited to, marching cubes, marching tetrahedrons, surface nets, greedy meshing, and double contour formation. In some configurations, the polygon 10759 can be generated by projecting the local neighborhoods of points along the normals of the points and connecting unconnected points. The resulting polygon 10759 can be based on, at least, the size of the neighborhoods, the maximum allowable distance with respect to the points to be considered, the maximum edge length with respect to the polygon, the minimum and maximum angles of the polygon, and the maximum deviation that the normals can take from each other. In some configurations, polygons 10759 can be filtered according to whether polygons 10759 would be too small for AV 10101 (Figure 1A) to pass through. In some configurations, circles the size of AV 10101 (Figure 1A) can be dragged around each of polygons 10759 by known means. If the circles substantially fit within polygons 10759, then polygons 10759, and therefore the resulting traversable surface, can accommodate AV 10101 (Figure 1A). In some configurations, the area of polygon 10759 can be compared to the area occupied by AV 10101 (Figure 1A). The polygons can be assumed to be irregular, and therefore the first step in determining the area of polygon 10759 is to separate polygon 10759 into regular polygons 10759A by known methods. For each regular polygon 10759A, a standard area equation can be used to determine its size. The areas of each regular polygon 10759A can be added together to determine the area of polygon 10759, which can then be compared to the area occupied by AV10101 (Figure 1A). The filtered polygons may include a subset of polygons that satisfy the size criteria. The filtered polygons can be used to define the final drivable surface.

Continuing to refer to Figure 1G, in some configurations, polygon 10759 can be processed by removing outliers using conventional means such as statistical analysis techniques, for example, but not limited to those available in Point Cloud Library, http://pointclouds.org/documentation/tutorials/statistical_outlier.php. Filtering may include reducing the size of segment 10137 (Figure 1D) by conventional means, including a voxelized grid approach, for example, those available in Point Cloud Library, http://pointclouds.org/documentation/tutorials/voxel_grid.php. The concave polygon 10263 can be created, for example, by a process described in, but is not limited to, "A New Concave Hull Algorithm and Concavenness Measure for n-dimensional Datasets," Park et al., Journal of Information Science and Engineering 28, pp. 587-600, 2012.

Referring primarily to Figure 1H, in some configurations, the processed point cloud data 10135 (Figure 1D) can be used to determine the initial drivable surface 10265. Region expansion can generate point clusters that may contain points that are part of the drivable surface. In some configurations, a reference plane can be fitted to each point cluster to determine the initial drivable surface. In some configurations, point clusters can be filtered according to the relationship between the orientation of the point cluster and the reference plane. For example, if the angle between the point cluster plane and the reference plane is, for example, less than approximately 30°, then the point cluster can be provisionally considered to be part of the initial drivable surface. In some configurations, point clusters can be filtered based on size constraints, for example, but not limited to size constraints. In some configurations, point clusters larger in point size than approximately 20% of the total points in the point cloud data 10131 (Figure 1D) can be considered too large, and point clusters smaller in size than approximately 0.1% of the total points in the point cloud data 10131 (Figure 1D) can be considered too small. The initial traversable surface may include filtered point clusters. In some configurations, point clusters can be divided by any of several known methods to continue further processing. In some configurations, density-based spatial clustering of noisy applications (DBSCAN) can be used to divide point clusters, while in some configurations, k-means clustering can be used to divide point clusters. DBSCAN can group points that are densely clustered together and mark points that are substantially isolated or in low-density regions as outliers. To be considered densely clustered, points must be located within a pre-selected distance from candidate points. In some configurations, a scale factor with respect to the pre-selected distance can be determined empirically or dynamically. In some configurations, the scale factor may be in the range of approximately 0.1 to 1.0.

Referring primarily to Figure 1I, generating an SDSF line (10163) (Figure 1D) may include locating the SDSF by further filtering of the concave polygon 10263 on the drivable surface 10265 (Figure 1H). In some configurations, points from the point cloud data constituting the polygon can be categorized as either an upper donut point 10351 (Figure 1J), a lower donut point 10353 (Figure 1J), or a cylindrical point 10355 (Figure 1J). The upper donut point 10351 (Figure 1J) may correspond to the shape of the SDSF model 10352 furthest from the ground. The lower donut point 10353 (Figure 1J) may correspond to the shape of the SDSF model 10352 closest to the ground or at ground level. The cylindrical point 10355 (Figure 1J) can correspond to a shape between the upper donut point 10351 (Figure 1J) and the lower donut point 10353 (Figure 1J). Combinations of categories can form a donut 10371. Certain criteria are tested to determine whether a donut 10371 forms an SDSF. For example, in each donut 10371, there must be a minimum number of points that are upper donut points 10351 (Figure 1J) and a minimum number that are lower donut points 10353 (Figure 1J). In some configurations, the minimum values can be selected empirically and can correspond to a range of about 5 to 20. Each donut 10371 can be divided into multiple parts, for example, into two hemispheres. Another criterion for determining whether a point within the donut 10371 represents an SDSF is whether the majority of the points are located within the opposing hemispheres of the portion of the donut 10371. The cylindrical point 10355 (Figure 1J) can occur in either the first cylindrical region 10357 (Figure 1J) or the second cylindrical region 10359 (Figure 1J). Another criterion for SDSF selection is that a minimum number of points must exist within both cylindrical regions 10357/10359 (Figure 1J). In some configurations, the minimum number of points can be empirically selected and may range from 3 to 20. Another criterion for SDSF selection is that the donut 10371 must contain at least two points from three categories: upper donut point 10351 (Figure 1J), lower donut point 10353 (Figure 1J), and cylindrical point 10355 (Figure 1J).

Continuing, primarily with reference to Figure 1I, in some configurations, polygons can be processed in parallel. Each category worker 10362 can search for its assigned polygon with respect to an SDSF point 10789 (Figure 1N) and assign the SDSF point 10789 (Figure 1N) to category 10763 (Figure 1G). As the polygons are processed, the resulting point categories 10763 (Figure 1G) can be combined (10363) to form a combined category 10366, and the categories can be shortened (10365) to form a shortened combined category 10368. Shortening the SDSF points 10789 (Figure 1N) may include filtering the SDSF points 10789 (Figure 1N) with respect to their distance from the ground. The shortened combined categories 10368 are averaged by exploring the area around each SDSF point 10766 (Figure 1G) and generating an average point 10765 (Figure 1G), which may be processed in parallel by an average worker 10373, so that the points of the categories can form a set of averaged donuts 10375. In some configurations, the radius around each SDSF point 10766 (Figure 1G) can be determined empirically. In some configurations, the radius around each SDSF point 10766 (Figure 1G) can range from 0.1m to 1.0m. The height change between one point and another on the SDSF trajectory 10377 (Figure 1G) with respect to the SDSF at the average point 10765 (Figure 1G) can be calculated. Connecting the averaged donuts 10375 together can generate the SDSF trajectory 10377 (Figures 1G and 1K). When creating the SDSF trajectory 10377 (Figures 1G and 1K), if two next candidate points exist within the search radius of the starting point, the next point can be selected based on the principle that it forms a line as straight as possible between the previous line segment, the starting point, and the candidate destination point, and then the candidate next point represents the smallest change in SDSF height between the previous point and the candidate next point. In some configurations, the SDSF height can be defined as the difference between the heights of the upper donut 10351 (Figure 1J) and the lower donut 10353 (Figure 1J).

Referring primarily to Figure 1L, combining a concave polygon and an SDSF line 10165 (Figure 1D) can generate a dataset including polygons 10139 (Figure 1D) and SDSF 10141 (Figure 1D), which can be manipulated to generate a graphed polygon using the SDSF data. Manipulating the concave polygon 10263 may include, but are not limited to, merging the concave polygon 10263 to form a merged polygon 10771. Merging the concave polygon 10263 can be carried out using known methods, such as, but are not limited to, those found at (http://www.angusj.com/delphi/clipper.php). The merged polygon 10771 may be smoothed at the edges and expanded to form an extended polygon 10772. The expanded polygon 10772 can be contracted to provide a travel margin, forming a contracted polygon 10774, to which an SDSF track 10377 (Figure 1M) can be added. Inward trimming (contraction) can ensure that there is room for AV 10101 (Figure 1A) to travel near the edges by reducing the size of the traversable surface by at least a pre-selected amount based on the size of AV 10101 (Figure 1A). Polygonal expansion and contraction can be performed by commercially available techniques such as, for example, the ARCGIS® clipping command (http://desktop.arcgis.com/en/arcmap/10.3/manage-data/editing-existing-features/clipping-a-polygon-feature.htm), for example.

Referring primarily to Figure 1M, the contracted polygon 10774 can be partitioned into polygons 10778, each of which can be traversed without encountering a non-traversable surface. The contracted polygon 10774 can be partitioned by conventional means such as ear slicing, which are optimized, for example, by z-order curve hashing and extended to handle holes, torsional polygons, degeneracy, and self-intersections. Commercially available ear slicing implementations include, but are not limited to, those found at (https://github.com/mapbox/earcut.hpp). The SDSF trajectory 10377 may include an SDSF point 10789 (Figure 1N) that can be connected to a polygon vertex 10781. Vertex 10781 can be considered a possible path point that can be interconnected to form a possible progression path for AV 10101 (Figure 1A). In the dataset, SDSF point 10789 (Figure 1N) can be labeled in this way. As partitioning proceeds, it is possible that redundant edges such as edges 10777 and 10779 may be introduced, for example, but not limited to these. Removing one of edges 10777 or 10779 can reduce the complexity of further analysis and allow for the retention of the polygonal mesh. In some configurations, the Hertel-Mehlhorn polygonal partitioning algorithm can be used to remove edges and omit edges that have been labeled as features. The set of polygons 10778, including the labeled features, can undergo further simplification to reduce the number of possible path points, which can then be provided to the device controller 10111 (Figure 1A) in the form of annotated point data 10379 (Figure 5B), which can be used to capture the occupied grid.

Referring here to Figure 2A-2B, the sensor data collected by the AV can also be used to capture the occupied grid. The processor in the AV can receive data from sensors in the long-range sensor assembly 20400 mounted on top of the cargo container 20110, and from short-range sensors 20510, 20520, 20530, 20540 located within the cargo platform 20160, as well as from other sensors. In addition, the processor may receive data from an optional short-range sensor 20505 mounted near the upper front of the cargo container 20110. The processor may also receive data from one or more antennas 20122A, 20122B (Figure 1C), including cellular, Wi-Fi, and/or GPS. In one embodiment, the AV20100 has a GPS antenna 20122A (Figure 1C) located on the long-range sensor assembly 20400 and/or an antenna 20122B (Figure 1C) located on the cargo container 20110. The processor may be located anywhere within the AV20100. In some embodiments, one or more processors are located within the long-range sensor assembly 20400. An additional processor may be located within the cargo platform 20160. In other embodiments, the processor may be located within the cargo container 20110 and/or as part of the power base 20170.

Continuing to refer to Figures 2A-2B, the long-range sensor assembly 20400 is mounted on top of a cargo container and provides an improved view of the environment surrounding the AV. In one embodiment, the long-range sensor assembly 20400 is positioned above 1.2 m feet from the surface of travel or the ground. In other embodiments, if the cargo container is higher, or if the power base configuration raises the cargo platform 20160, the long-range sensor assembly 20400 may be positioned 1.8 m from the ground over which the AV is traveling. The long-range sensor assembly 20400 provides information about the environment surrounding the AV from a minimum distance to a maximum range. The minimum distance may be defined by the relative positions of the long-range sensor 20400 and the cargo container 20110. The minimum distance may further be defined by the field of view (FOV) of the sensor. The maximum distance may be defined by the range of the long-range sensor within the long-range sensor assembly 20400 and/or by the processor. In one embodiment, the range of the long-range sensor is limited to 20 meters. In another embodiment, the Velodyne Puck Lidar has a range of up to 100 meters. The long-range sensor assembly 20400 may provide data on objects in all directions. The sensor assembly may provide information on structures, surfaces, and obstacles across a 360° angle around the AV20100.

Continuing to refer to Figure 2A, three long-range cameras observed through windows 20434, 20436, and 20438 can provide horizontal FOVs 20410, 20412, and 20414, all of which provide a 360° FOV. The horizontal FOV may be defined by the selected camera and the camera's location within the long-range camera assembly 20400. In describing the field of view, the zero angle is a ray of light located in a vertical plane perpendicular to the front of the AV, passing through the center of the AV20100. The zero-angle ray passes through the front of the AV. The front long-range camera viewed through window 20434 has a 96° FOV 20410, ranging from 311° to 47°. The left-side long-range camera viewed through window 20436 has an FOV 20412, ranging from 47° to 180°. The right-side long-range camera, viewed through window 20438, has an FOV of 180° to 311°. The long-range sensor assembly 20400 may also include an industrial camera positioned to observe through window 20432, providing more detailed information about objects and surfaces in front of the AV 20100 than the long-range camera. The industrial camera located behind window 20432 may have an FOV of 20416, defined by the selected camera and the camera's location within the long-range camera assembly 20400. In one embodiment, the industrial camera behind window 20432 has an FOV of 23° to 337°.

Referring to Figure 2B, the LIDAR 20420 provides a 360° horizontal FOV around the AV20100. The vertical FOV may be limited by the LIDAR device. In one embodiment, a vertical FOV 20418, with a 40° angle and mounted 1.2m to 1.8m above the ground, sets the minimum sensor distance from the AV20100 to 3.3m to 5m.

Referring here to Figures 2C and 2D, the long-range sensor assembly 20400 is shown together with the cover 20430. The cover 20430 includes windows 20434, 20432, and 20436 through which the long-range camera and industrial camera observe the environment surrounding the AV 20100. The cover 20430 for the long-range sensor assembly 20400 is sealed from the weather by an O-ring between the cover 20430 and the top of the cargo container 20110.

Referring here to Figures 2E and 2F, cover 20430 has been removed to expose an embodiment of the camera and processor. Lidar sensor 20420 provides data relating to the range or distance to the surrounding surface of the AV. This data may be provided to processor 20470, located within long-range sensor assembly 20400. The Lidar is mounted on long-range cameras 20440A-C and structure 20405 above cover 20430. Lidar sensor 20420 is an embodiment of a ranging sensor based on reflected laser pulsed light. Other ranging sensors, such as radar, which use reflected radio waves, can also be used. In one embodiment, Lidar sensor 20420 is a Puck sensor by VELODYNE Lidar® (San Jose, CA). Three long-range cameras 20440A, 20440B, and 20440C provide digital images of objects, surfaces, and structures surrounding the AV20100. The three long-range cameras 20440A, 20440B, and 20440C are arranged around the structure 20405 relative to the cover 20430 to provide three horizontal FOVs covering the entire 360° around the AV. The long-range cameras 20440A, 20440B, and 20440C are mounted on the lifting ring structure 20405, which is mounted on the cargo container 20110. The long-range cameras 20440A, 20440B, and 20440C receive images through windows 20434, 20436, and 20438 mounted within the cover 20430. The long-range cameras may comprise a camera and lens on a printed circuit board (PCB).

Referring to Figure 2F, one embodiment of the long-range camera 20440A may include a digital camera 20444 with a fisheye lens 20442 mounted on the front of the digital camera 20444. The fisheye lens 20442 can greatly widen the camera's field of view (FOV). In one embodiment, the fisheye lens expands the field of view to 180°. In one embodiment, the digital camera 20444 is similar to the e-cam52A_56540_MOD by E-con Systems (San Jose, CA). In one embodiment, the fisheye lens 20442 is similar to the Model DSL227 by Sunex (Carlsbad, CA).

Continuing to refer to Figure 2F, the long-range sensor assembly 20400 may also include an industrial camera 20450 that receives visual data through a window 20432 in the cover 20430. The industrial camera 20450 provides the processor 20470 with additional data regarding objects, surfaces, and structures in front of the AV. The camera may be similar to a Kowa industrial camera (part number LM6HC). The industrial camera 20450 and the long-range cameras 20440A-C are positioned 1.2m to 1.8m above the surface over which the AV 20100 is moving.

Continuing to refer to Figure 2F, mounting the long-range sensor assembly 20400 on top of a cargo container offers at least two advantages. The field of view for the long-range sensor, including the long-range cameras 20440A-C, the industrial camera 20450, and the LIDAR 20420, is often less obstructed by nearby objects such as people, vehicles, and low walls when the sensor is mounted higher above the ground. In addition, pedestrian walkways are designed to provide visual cues, including signs and fence heights, for people to perceive, with a typical eye level ranging from 1.2m to 1.8m. Mounting the long-range sensor assembly 20400 on top of a cargo container places the long-range cameras 20440A-C and 20450 at the same level as signs and on visual cues directed towards pedestrians. The long-range sensor is mounted on a structure 20405 that provides a substantially rigid mount to withstand the deflection caused by the movement of the AV20100.

Referring again to Figures 2E and 2F, the long-range sensor assembly may include an inertial measurement unit (IMU) and one or more processors that receive data from the long-range sensor and output the processed data to other processors for navigation. An IMU 20460 with a vertical reference unit (VRU) is mounted on structure 20405. The IMU/VRU 20460 may be positioned directly below the LIDAR 20420 to provide positional data related to the LIDAR 20420. Position and orientation data from the IMU/VRU 20460 may be combined with data from other long-range sensors. In one embodiment, the IMU/VRU 20460 is model MTi 20 supplied by Xsens Technologies (The Netherlands). One or more processors may include processor 20465 that receives data from at least the industrial camera 20450. In addition, the processor 20470 may receive data from at least one of the following: the LiDAR 20420, the long-range cameras 20440A-C, the industrial camera 20450, and the IMU/VRU 20460. The processor 20470 may be cooled by a liquid-cooled heat exchanger 20475 connected to a circulating coolant system.

Referring here to Figure 2G, the AV 20100 may include several short-range sensors that detect the travel surface and obstacles within a predetermined distance from the AV. The short-range sensors 20510, 20520, 20530, 20540, 20550, and 20560 are located above the perimeter of the container platform 20160. These sensors are located below the cargo container 20110 (Figure 2B) and are closer to the ground than the long-range sensor assembly 20400 (Figure 2C). The short-range sensors 20510, 20520, 20530, 20540, 20550, and 20560 are angled downward to provide an FOV that captures surfaces and objects that cannot be seen by the sensors in the long-range sensor assembly 20400 (Figure 2C). A sensor positioned closer to the ground and angled downwards has a field of view that is less likely to be obstructed by nearby objects and pedestrians than a sensor mounted further from the ground. In one embodiment, the short-range sensor provides information about the ground surface and objects up to 4 meters from AV20100.

Referring again to Figure 2B, the vertical FOVs for two of the short-range sensors are shown in the side view of AV 20100. The vertical FOV 20542 of the rear-facing sensor 20540 is centered on the centerline 20544. The centerline 20544 is angled downwards from the upper surface of the cargo platform 20160. In one embodiment, sensor 20540 has a vertical FOV of 42° and a centerline 20546 angled 22° to 28° downwards from the plane 20547 defined by the upper plate of the cargo platform 20160. In one embodiment, the short-range sensors 20510 and 20540 are located approximately 0.55 m to 0.71 m above the ground. The resulting FOVs 20512 and 20542 cover the ground from 0.4 m to 4.2 m above the AV. The short-range sensors 20510, 20520, 20530, 20550 (Figure 2G), and 20560 (Figure 2G), mounted on the cargo base 20160, have similar vertical field of view and centerline angles relative to the top of the cargo platform. The short-range sensors mounted on the cargo platform 20160 can perceive the ground from 0.4 to 4.7 meters from the outer edge of the AV20100.

Continuing with Figure 2B, the short-range sensor 20505 may be mounted on the front of the cargo container 20110, near the top. In one embodiment, sensor 20505 may provide an additional view of the ground in front of the AV, in addition to the view provided by short-range sensor 20510. In another embodiment, sensor 20505 may provide a view of the ground in front of the AV instead of the view provided by short-range sensor 20510. In one embodiment, short-range sensor 20505 may have a vertical FOV 20507 of 42°, and the angle of its centerline relative to the top of cargo platform 20160 is 39°. The resulting ground view extends from 0.7m to 3.75m from the AV.

Referring again to Figure 2G, the horizontal FOVs of the short-range sensors 20510, 20520, 20530, 20540, 20550, and 20560 cover all directions around the AV 20100. The horizontal FOVs 20522 and 20532 of adjacent sensors such as 20520 and 20530 overlap at a certain distance from the AV 20100. In one embodiment, the horizontal FOVs 20522, 20532, and 20562 and 20552 of adjacent sensors 20520, 20530, and 20560, and 20550 overlap at 0.5 to 2 meters from the AV. The short-range sensors are distributed around the cargo base 20160, have a horizontal field of view, are installed at specific angles, and provide nearly complete visual coverage of the ground surrounding the AV. In one embodiment, the short-range sensor has a horizontal FOV of 69°. The front sensor 20510 faces forward at zero angle to the AV and has an FOV of 20512. In one embodiment, the two front corner sensors 20520, 20560 are angled such that their centerlines are at an angle of 65° 20564. In another embodiment, the rear sensors 20530, 20550 are angled such that their centerlines are at an angle of 110° 20534. In some configurations, a number of other sensors with other horizontal FOVs, mounted around the cargo base 20160 to provide a nearly complete view of the ground around the AV 20100, are also possible.

Referring to Figure 2H, the short-range sensors 20510, 20520, 20530, 20540, 20550, and 20560 are positioned around the cargo base 20160. The short-range camera is mounted within a projection that sets the angle and position of the short-range camera. In another configuration, the sensors are mounted and positioned on the interior of the cargo base and receive visual data through a window aligned with the outer casing of the cargo base 20160.

Referring here to Figures 2I and 2J, the short-range sensor 20600 is mounted within an outer panel element 20516 of the cargo base 20160 and may include a liquid cooling system. The outer panel element 20516 includes a formed projection 20514 that holds the short-range sensor assembly 20600 at a predetermined location and vertical angle to the top of the cargo base 20160 and at a certain angle to the front of the cargo base 20160. In some configurations, the short-range sensor 20510 is angled downward by 28° relative to the cargo platform 20160, the short-range sensors 20520 and 20560 are angled 18° downward and 25° forward, the short-range sensors 20530 and 20550 are angled 34° downward and 20° backward, and the short-range sensor 20540 is angled downward by 28° relative to the cargo platform 20160. The outer panel element 20516 includes a cavity 20517 for receiving the camera assembly 20600. The outer panel element 20516 may also include a plurality of elements 20518 for receiving mechanical fasteners, including, but not limited to, rivets, screws, and buttons. Alternatively, the camera assembly may be mounted using adhesive or held in place using clips fastened to the outer panel element 20516. A gasket 20519 can provide a seal to the front of the camera 20610.

Referring here to Figures 2K and 2L, the short-range sensor assembly 20600 comprises a short-range sensor 20610 mounted on a bracket 20622 attached to a water-cooled plate 20626. The outer case 20612, transparent cover 20614, and heat sink 20618 are partially removed in Figures 2K and 2L to better visualize the sensor block 20616 and electronic block 20620, which are heat dissipation elements for the short-range sensor 20610. The short-range sensor assembly 20600 may include one or more thermoelectric coolers (TECs) 20630 between the bracket 20622 and the water-cooled plate 20626. The water-cooled plate 20626 is cooled by a coolant that is pumped through 20628 which is thermally connected to the plate 20626. The TEC is an electrically powered element with first and second sides. The electric TEC cools the first side while rejecting the thermal energy removed from the first side, plus electrical energy, on the second side. In the short-range sensor assembly 20600, the TEC 20630 cools the bracket 20622 and transfers the thermal cooling energy plus electrical energy to the water-cooled plate 20626. Alternatively, the TEC 20630 can be used to actively control the temperature of the camera 20600 by varying the magnitude and polarity of the voltage supplied to the TEC 20630.

Operating the TEC 20630 in cooling mode allows the short-range sensor 20610 to operate at temperatures below the coolant temperature. The bracket 20622 is thermally connected to the short-range sensor 20610 in two locations to maximize cooling of the sensor block 20616 and the electronic block 20620. The bracket 20622 includes a tab 20624, which is thermally attached to the heatsink 20618 via a screw 20625. The heatsink 20618 is thermally connected to the sensor block 20616. The bracket is therefore thermally connected to the sensor block 20616 via the heatsink 20618, the screw 20625, and the tab 20624. The bracket 20622 is also mechanically attached to the electronic block 20620, providing direct cooling of the electronic block 20620. Bracket 20622 may include, but is not limited to, multiple mechanical attachments, including screws and rivets that engage with element 20518 in Figure 2J. The short-range sensor 20610 may incorporate, but is not limited to, one or more sensors, including cameras, stereo cameras, ultrasonic sensors, short-range radars, infrared projectors, and CMOS sensors. An exemplary short-range sensor is similar to the RealSense depth camera D435 by Intel (Santa Clara, California), which comprises an IR projector, two imager chips, and an RGB camera.

Referring here to Figure 2M-2O, another embodiment of the AV20100 is shown. The AV20100A includes a cargo container 20110 mounted on a cargo platform 20160 and a power base 20170. The AV20100A includes a plurality of long-range and short-range sensors. The primary long-range sensor is mounted in a sensor pylon 20400A on the cargo container 20110. The sensor pylon may include a LIDAR 20420 and a plurality of long-range cameras (not shown) oriented in a divergent direction to provide a wide field of view. In some configurations, the LIDAR 20420 can be used, for example, to provide point cloud data that may enable the capture of an occupied grid, to identify landmarks, to locate the AV20100 in its environment, and/or to provide information for determining navigable space, as described elsewhere in this specification. In some configurations, Leopard Imaging Inc. Long-range cameras can be used to identify landmarks, locate the AV20100 within its environment, and/or determine navigable space.

Continuing to refer to Figure 2M-2O, the short-range sensors are primarily mounted within the cargo platform 20160 and provide information about obstacles in the vicinity of AV20100A. In some embodiments, the short-range sensors supply data about obstacles and surfaces within 4 m of AV20100A. In some configurations, the short-range sensors provide information up to 10 m from AV20100A. At least partially, multiple forward-facing cameras are mounted within the cargo platform 20160. In some configurations, the multiple cameras may include three cameras.

Referring here to Figure 2O, the upper cover 20830 is partially cut away to expose the sub-roof 20810. The sub-roof 20810 provides a single component on which multiple antennas 20820 can be mounted. In one embodiment, ten antennas 20820 are mounted on the sub-roof 20810. Furthermore, for example, there are four cellular communication channels, each having two antennas, and two Wi-Fi antennas. The antennas are wired as primary and auxiliary antennas for cellular transmission and reception. The auxiliary antennas can improve cellular functionality in several ways, including, but not limited to, reducing interference and achieving 4G LTE connectivity. The sub-roof 20810 and the upper cover 20830 are non-metallic. The sub-roof 20810 is a plastic surface within 10-20 mm of the upper cover 20830, which is non-structural and allows the antenna to be connected to the processor before the upper cover 20830 is installed. Antenna connections are often high impedance and sensitive to dirt, grease, and misuse. Mounting and connecting the antenna to the sub-roof 20810 allows the upper cover to be installed and removed without touching the antenna connections. Maintenance and repair operations may include removing the upper cover without removing the sub-roof or disconnecting the antenna. Assembling the antenna separately from the installation of the upper cover 20830 facilitates testing/repair. The upper cover 20830 is weatherproof and prevents water and dust from entering the cargo container 20110. Mounting the antenna on the sub-roof minimizes the number of openings on the upper cover 20830.

Referring here to Figures 2P, 2Q, and 2R, another embodiment of the Long Range Sensor Assembly (LRSA) 20400A, mounted on a cargo container (not shown), is shown. The LRSA may include a LiDAR and multiple long-range cameras, mounted at different positions on the LRSA structure 20950 to provide a panoramic view of the environment of the AV20100A. The LiDAR 20420 is mounted at the very top of the LRSA structure 20950 to provide an uninterrupted view. The LiDAR may include a Velodyne LiDAR. Multiple long-range cameras 20910A–20910D are mounted on the LRSA structure 20950 on the next level below the LiDAR 20420. In one embodiment, four cameras are mounted one at each 90° interval around the structure to provide four views of the environment around the AV20100. In some embodiments, the four views will overlap. In some embodiments, each camera is either aligned with the direction of motion or perpendicular to the direction of motion. In one embodiment, one camera aligns with each of the main faces of the AV20100A, i.e., the front, rear, left, and right sides. In one embodiment, the long-range camera is a model LI-AR01 44-MIPI-M12 manufactured by Leopard Imaging Inc. The long-range camera may have a MIPI CSI-2 interface to provide high-speed data transfer to the processor. The long-range camera may have a horizontal field of view of 50°–70° and a vertical field of view of 30°–40°.

Referring here to Figures 2S and 2T, the long-range processor 20940 is located on the LRSA structure 20950 below the long-range cameras 20910A-20910D and the LiDAR 20420. The long-range processor 20940 receives data from the long-range cameras and LiDAR. The long-range processor communicates with one or more processors located elsewhere in the AV20100A. The long-range processor 20940 provides the data derived from the long-range cameras and LiDAR to one or more processors located elsewhere on the AV20100A, as described elsewhere in this specification. The long-range processor 20940 may be liquid-cooled by a cooler 20930. The cooler 20930 may be mounted in the structure below the long-range cameras and LiDAR. The cooler 20930 may provide a mounting location for the long-range processor 20940. Cooler 20930 is described in U.S. Patent Application No. 16/883,668 (Patent Attorney No. AA280), filed on 26 May 2020, titled "Apparatus for Electronic Cooling on an Autonomous Device" (incorporated herein by reference as a whole). The cooler is provided with a liquid supply conduit and a return conduit for supplying a cooling liquid to cooler 20930.

Referring again to Figures 2M and 2N, the short-range camera assembly 20740A-C is mounted on the front of the container platform 20160 and is angled to collect information about the surface underway and obstacles, steps, curbs, and other substantial discontinuous surface features (SDSFs). The camera assembly 20740A-C includes one or more LED lights to illuminate the surface underway, objects on the ground, and SDSFs.

Referring here to Figure 2U-2X, camera assemblies 20740A-B include a light 20732 to illuminate the ground and objects and provide improved image data from camera 20732. Note that camera assembly 20740A is a mirror image of 20740C, and the description of 20740A implicitly applies to 20740C. Camera 20732 may include a single-focus camera, a stereo camera, and/or an infrared projector and a CMOS sensor. One embodiment of the camera is the Real-Sense Depth D435 camera by Intel (Santa Clara, CA), which includes an IR projector, two imager chips with lenses, and an RGB camera. The LED light 20734 may be used at night or in low-light conditions, or may be used continuously to improve the image data. One operating principle is that the light creates contrast by illuminating the projection surface and casting shadows in recesses. In one embodiment, the LED light may be a white LED. In one embodiment, the LED light 20734 is an Xlamp XHP50 from Cree Inc. In another embodiment, the LED light may emit infrared light to provide illumination for the camera 20372 without distracting or disturbing nearby pedestrians or drivers.

Continuing to refer to Figure 2U-2X, the placement and angle of light 20374 and the shape of covers 20736A and 20736B prevent camera 20732 from viewing light 20374. The angle and placement of light 20374 and covers 20736A and 20736B prevent the light from interfering with drivers or obstructing pedestrians. It is advantageous that camera 20732 is not exposed to light 20734 to prevent the sensor within camera 20732 from being blinded by light 20734 and thus being prevented from detecting weaker light signals from the ground and objects in front of and to the sides of AV20100A. Camera 20732 and/or light 20734 may be cooled using a liquid flowing in and out of the camera assembly through port 20736.

Referring here to Figures 2W and 2X, the short-range camera assembly 20740A includes an ultrasonic or sonar short-range sensor 20730A. The second short-range camera assembly 20740C also includes an ultrasonic short-range sensor 20730B (Figure 2N).

Referring to Figure 2Y, the ultrasonic sensor 20730A is mounted above the camera 20732. In one embodiment, the centerline of the ultrasonic sensor 20730A is parallel to the base of the cargo container 20110, which often means that the sensor 20730A is horizontal. The sensor 20730A is angled 45° from a forward-facing position. The cover 20376A provides a horn 20746 for directing the ultrasonic waves emerging from and received by the ultrasonic sensor 20730A.

Continuing to refer to Figure 2Y, the cross-sections of camera 20732 and light 20734 within camera assembly 20740A illustrate the angles and openings in cover 20736. The cameras in the short-range camera assembly are angled downward to better image the ground in front of and to the sides of the AV. The center camera assembly 20740B is oriented directly forward in the horizontal plane. The corner camera assemblies 20740A and 20740C are angled 25° to their respective sides relative to directly forward in the horizontal plane. Camera 20732 is angled 20° downward relative to the top of the cargo platform in the vertical plane. Since the AV generally keeps the cargo platform horizontal, the cameras are therefore angled 20° downward from the horizontal. Similarly, the center camera assembly 20740B (Figure 2M) is angled 28° downward from the horizontal. In one embodiment, the camera in the camera assembly may be angled downward by 25° to 35°. In another embodiment, the camera in the camera assembly may be angled downward by 15° to 45°. The LED light 20734 is similarly angled downward to illuminate the ground imaged by the camera 20732 and to minimize distraction to pedestrians. In one embodiment, the LED light centerline 20742 is parallel to the camera centerline 20738 within 5°. The cover 20736A protects both the camera 20732 and pedestrians from the bright light of the LED 20734 in the camera assembly 20740A-C. The cover, which isolates the light emitted by the LED, also provides a gradually widening opening 20737 to maximize the field of view of the camera 20732. The light is embedded at least 4 mm from the opening in the cover. The light opening is defined by the upper wall 20739 and the lower wall 20744. The upper wall 20739 is approximately parallel (±5°) to the centerline 2074. The lower wall 20744 gradually widens to approximately 18° from the centerline 20742 to maximize illumination of the ground and objects near the ground.

Referring here to Figure 2Z-2AA, in one configuration, each light 20734 includes two LEDs 20734A under a square lens 20734B to generate a beam of light. The LEDs/lenses are angled and positioned relative to the camera 20372 to illuminate the camera's field of view with minimal light spill outside the camera's FOV. The two LEDs/lenses are mounted together on a single PCB 20752 with a defined angle 20762 between the two lights. In another configuration, the two LEDs/lenses are mounted individually on separate PCBs with heatsinks 20626A at a certain angle to each other. In one embodiment, the light is an Xlamp XHP50 from Cree, Inc., and the lens is a 60° lens HB-SQ-W from LEDil. The lights are angled approximately 50° relative to each other, and therefore the angle between the front of the lenses is 130°. The lights are located approximately 18 mm ( ± 5 mm) rear of the front of the camera 20732 and approximately 30 mm below the centerline of the camera 20732.

Referring here to Figure 2AA-2BB, camera 20732 is cooled by a thermoelectric cooler (TEC) 20630, which, along with light 20734, is cooled by a liquid coolant flowing through a cryogenic block 20626A. The camera is mounted to bracket 20622 via screws 20625 that screw into the sensor block portion of the camera, while the rear of bracket 20622 is bolted to the camera's electronics block. Bracket 20622 is cooled by two TECs to maintain the performance of the IR imaging chip (CMOS chip) within camera 20732. The TECs dissipate the heat from bracket 20622 and the power they draw in to the cryogenic block 20626A.

Referring to Figure 2BB, the coolant is directed through a U-shaped path created by the central fin 20626D. The coolant flows directly behind the LED/lens/PCB of light 20734. Fins 20626B and 20626C improve heat transfer from light 20734 to the coolant. The coolant flows upward, passing alongside the high-temperature side of TEC 20630. The fluid path is created by plate 20737 (Figure 2X), which is attached to the rear of the low-temperature block 20626A.

Referring to Figure 3A, sensor data and map data can be used to update the occupied grid. The system and method of this teaching can manage a global occupied grid for a device autonomously navigating with respect to a grid map. The grid map may include routes or paths that the device can take from a starting point to a destination. The global occupied grid may include free-space indications that indicate where it is safe for the device to navigate. Possible routes and free-space indications can be combined on the global occupied grid to establish the optimal path that the device can take to safely reach its destination.

Continuing to refer to Figure 3A, the global occupancy grid, which will be used to determine a non-interference navigation route as the device moves, can be accessed based on the device's location, and the global occupancy grid can be updated as the device moves. The update can be based on at least the current value associated with the global occupancy grid at the device's location, a static occupancy grid which may contain historical information about the neighborhood the device is navigating, and data collected by sensors as the device moves. Sensors can be located on the device as described herein, and they can be located elsewhere.

Continuing to refer to Figure 3A, the global occupancy grid can contain cells, and each cell can be associated with an occupancy probability value. Each cell in the global occupancy grid can be associated with information such as the characteristics and discontinuities of the surrounding traversal surface, as well as prior occupancy data associated with that location, as determined from previously collected data and from data collected as the device navigates, as well as whether an obstacle has been identified at the cell's location. Data captured as the device navigates can be stored in a local occupancy grid, whose center is the device. When updating the global occupancy grid, static, previously collected data can be combined with the local and global occupancy grid data determined in the previous update to create a new global occupancy grid with spaces occupied by the device that are marked as unoccupied. In some configurations, a Bayesian method can be used to update the global occupancy grid. This method may include, for each cell in the local occupancy grid, calculating the cell's position on the global occupancy grid, accessing the value at that position from the current global occupancy grid, accessing the value at that position from the static occupancy grid, accessing the value at that position from the local occupancy grid, and calculating a new value at that position on the global occupancy grid as a function of the current value from the global occupancy grid, the value from the static occupancy grid, and the value from the local occupancy grid. In some configurations, the relationship used to calculate the new value may include subtracting the current value from the sum of the static value and the local occupancy grid value. In some configurations, the new value may be limited by a pre-selected value, for example, based on computational limits.

Continuing to refer to Figure 3A, the system 30100 of this teaching can manage a global occupancy grid. The global occupancy grid can start with initial data and can be updated as the device moves. Creating the initial global occupancy grid may include a first process, and updating the global occupancy grid may include a second process. The system 30100 may include a global occupancy server 30121 that can receive information from various sources and update the global occupancy grid 30505 based on at least the information. The information may be supplied by sensors located on and/or elsewhere, static information, and navigation information, for example, but not limited to. In some configurations, the sensors may include cameras and radar, for example, that can detect surface characteristics and obstacles. The sensors may be positioned on the device to provide sufficient peripheral coverage to enable safe movement by the device, for example. In some configurations, the LIDAR 30103 can provide LIDAR point cloud (PC) data 30201, which may enable the capture of a locally occupied grid using LIDAR free-space information 30213. In some configurations, a conventional ground detection inverse sensor model (ISM) 30113 can process the LIDAR PC data 30201 and generate LIDAR free-space information 30213.

Continuing to refer to Figure 3A, in some configurations, the RGB-D camera 30101 can provide RGB-D PC data 30202 and RGB camera data 30203. The RGB-D PC data 30202 can capture a local occupied grid using depth free space information 30209, and the RGB-D camera data 30203 can capture a local occupied grid using surface data 30211. In some configurations, the RGB-D PC data 30202 can be processed by, for example, a conventional stereo free space ISM 30109, and the RGB-D camera data 30203 can be fed into, for example, a conventional surface detection neural network 30111, for example. In some configurations, the RGB MIPI camera 30105 provides RGB data 30205, which, combined with LIDAR PC data 30201, can generate a locally occupied grid with LIDAR/MIPI free-space information 30215. In some configurations, the RGB data 30205 can be fed into a conventional free-space neural network 30115, whose output, along with the LIDAR PC data 30201, can receive a pre-selected mask 30221 that identifies the most accuracy-critical portion of the RGB data 30205 before being fed into a conventional 2D-3D alignment 30117. The 2D-3D alignment 30117 can project an image from the RGB data 30205 onto the LIDAR PC data 30201. In some configurations, the 2D-3D alignment 30117 is not required. Any combination of sensors and methods for processing sensor data can be used to collect data and update the global occupancy grid. Any number of free-space estimation procedures can be used and combined to enable the determination and verification of occupancy probabilities within the global occupancy grid.

Continuing to refer to Figure 3A, in some configurations, historical data can be provided by a repository 30107 of previously collected and processed data, which, for example, has information associated with the navigation area. In some configurations, the repository 30107 may include, for example, route information such as polygons 30207, but is not limited to these. In some configurations, this data can be fed into a conventional polygon parser 30119, which can provide edges 30303, discontinuities 30503, and surfaces 30241 to a global occupancy grid server 30121. The global occupancy grid server 30121 can merge the locally occupy grid data collected by the sensors with the processed repository data to determine the global occupancy grid 30505. A grid map 30601 (Figure 3D) can be created from the global occupancy data.

Referring here to Figure 3B, in some configurations, the sensor may include a sonar 30141 that can provide a local occupied grid with sonar free space 30225 to a global occupied grid server 30121. Depth data 30209 can be processed by a conventional free-space ISM 30143. The local occupied grid with sonar free space 30225 can be merged with a local occupied grid with surface and discontinuity 30223, a local occupied grid with LIDAR free space 30213, a local occupied grid with LIDAR/MIPI free space 30215, a local occupied grid with stereo free space 30209, and with edges 30303 (Figure 3F), discontinuity 30503 (Figure 3F), navigation point 30501 (Figure 3F), surface confidence 30513 (Figure 3F), and surface 30241 (Figure 3F) to form a global occupied grid 30505.

Referring here to Figure 3C-3F, in order to initialize the global occupied grid, the global occupied grid initializer 30200 may include the global occupied grid server 30121 creating the global occupied grid 30505 and the static grid 30249. The global occupied grid 30505 can be created by fusing data from the local occupied grid 30118 with edges 30303, discontinuities 30503, and surfaces 30241 located within the area of interest. The static grid 30249 (Figure 3D) can be created to include, for example, surface data 30241, discontinuity data 30503, edges 30303, and polygon data 30207, etc. The initial global occupancy grid 30505 can be calculated by adding occupancy probability data from the static grid 30249 (Figure 3E) to occupancy data derived from data collected from sensor 30107A, and subtracting occupancy data from the prior global occupancy grid 30505A (Figure 3F). The local occupancy grid 30118 may include, but are not limited to, local occupancy grid data obtained from stereo free-space estimation 30209 (Figure 3B) via ISM, local occupancy grid data including surface/discontinuity detection results 30223 (Figure 3B), local occupancy grid data obtained from LIDAR free-space estimation 30213 (Figure 3B) via ISM, and, in some configurations, local occupancy grid data obtained from LIDAR/MIPI free-space estimation 30215 (Figure 3B) following 2D-3D alignment 30117 (Figure 3B). In some configurations, the local occupancy grid 30118 may include local occupancy grid data derived from sonar free-space estimation 30225 via ISM. In some configurations, various local occupancy grids with free-space estimations can be merged into the local occupancy grid 30118 according to a pre-selected known process. From the global occupancy grid 30505, a grid map 30601 (Figure 3E) can be created, which may include occupancy and surface data of the device's vicinity. In some configurations, the grid map 30601 (Figure 3E) and the static grid 30249 (Figure 3D) can be exposed using, for example, a Robot Operating System (ROS) subscribe/publish feature, but are not limited to these.

Referring here to Figures 3G and 3H, in order to update the occupied grid as the device moves, the occupied grid update 30300 may include updating the local occupied grid with respect to data measured as the device is moving and combining that data with the static grid 30249. The static grid 30249 is accessed when the device moves out of the occupied grid range it is working with. The device can be positioned in occupied grid 30245A at a first location 30513A in a first time. When the device moves to a second location 30513B, the device is positioned in occupied grid 30245B, which includes a set of values derived from the values in occupied grid 30245A from its new location. Data from the static grid 30249 and surface data from the initial global occupied grid 30505 (Figure 3C), which is spatially located to the cells in occupied grid 30245B in the second time, can be used together with measured surface data and occupancy probabilities to update each grid cell according to a pre-selected relationship. In some configurations, the relationship may include summing static data with measured data. The resulting occupied grid 30245C at a third time and third location 30513C can be made available to the movement manager 30123 to signal the navigation of this device.

Referring here to Figure 3I, the method 30450 for creating and managing a occupy grid may include, but is not limited to, converting sensor measurements into reference frames associated with the device by a local occupy grid creation node 30122 30451, creating a timestamped measurement occupy grid 30453, and exposing the timestamped measurement occupy grid as a local occupy grid 30234 (Figure 3G) 30455. The system associated with method 30450 may include multiple local grid creation nodes 30122, for example, one per sensor, so that multiple local occupy grids 30234 (Figure 3G) may be brought about. The sensors may include, but are not limited to, an RGB-D camera 30325 (Figure 3G), a LiDAR/MIPI 30231 (Figure 3G), and a LiDAR 30233 (Figure 3G). A system associated with Method 30450 may include a global occupancy grid server 30121 that can receive local occupancy grids and process them according to Method 30450. In particular, Method 30450 may include loading surfaces 30242, accessing surface discontinuities such as curbs 30504, and creating a static occupancy grid 30249 from any properties available in the repository 30107, which may include surfaces and surface discontinuities 30248. Method 30450 may include receiving published local occupancy grids 30456 and moving a global occupancy grid to keep the device at the center of the map 30457. Method 30450 may include setting up a new area on the map with prior information from a static pre-occupancy grid 30249 30459 and marking an area currently occupied by the device as unoccupied 30461. Method 30450 may execute loop 30463 for each cell in each local occupying grid. Loop 30463 may include, but is not limited to, calculating the cell's location on the global occupying grid, accessing the previous value at that location on the global occupying grid, and calculating a new value at the cell location based on the relationship between the previous value and the value at the cell in the local occupying grid. The relationship may include, but is not limited to, summing the values. Loop 30463 may include comparing the new value against a pre-selected range of acceptable probabilities and setting the global occupying grid with the new value. The comparison may include setting the probability to the minimum or maximum acceptable probability if the probability is lower or higher than the minimum or maximum acceptable probability. Method 30450 may also include exposing the global occupying grid 30467.

Referring here to Figure 3J, an alternative method 30150 for creating a global occupancy grid may include, but is not limited to, step 30151, if the device moves, accessing the occupancy probability value associated with the old map area (where the device was before it moved) and updating the global occupancy grid on the new map area (where the device was after it moved) using the value from the old map area 30153, accessing the drivable surface associated with the cells of the global occupancy grid in the new map area and updating the cells in the updated global occupancy grid using the drivable surface 30155, and proceeding to step 30159. In step 30151, if the device has not moved and the global occupancy grid is co-located with the local occupancy grid, method 30150 may include updating the global occupancy grid as possible using surface confidence associated with drivable surfaces from at least one local occupancy grid 30159, updating the global occupancy grid as updated using log odds of occupancy probability values from at least one local occupancy grid, for example, using a Bayesian function 30161, and adjusting the log odds based on at least a characteristic associated with its location 30163. In step 30157, if the global occupancy grid is not co-located with the local occupancy grid, method 30150 may include returning to step 30151. The characteristic may include, but is not limited to, setting the device's location as unoccupied.

Referring here to Figure 3K, in an alternative configuration, method 30250 for creating a global occupancy grid may, but is not limited to, 30251, update the global occupancy grid with information from a static grid associated with the device's new location 30253 if the device moves. Method 30250 may also include analyzing the surface at the new location 30257. 30259, if the surface is traversable, method 30250 may include updating the surface on the global occupancy grid 30261 and updating the global occupancy grid with values from a repository of static values associated with the new location on the map 30263.

Referring here to Figure 3L, updating the surface 30261 may include, but is not limited to, accessing the local occupied grid (LOG) for a particular sensor 30351. In 30353, if further cells to be processed reside within the local occupied grid, method 30261 may include accessing the surface classification confidence value and surface classification from the local occupied grid 30355. In 30357, if the surface classification of a cell in the local occupied grid is identical to the surface classification in the global occupied grid at the cell's location, method 30261 may include setting the new global occupied grid (GOG) surface confidence to the sum of the old global occupied grid surface confidence and the local occupied grid surface confidence 30461. In 30357, if the surface classification of a cell in the local occupied grid is not identical to the surface classification in the global occupied grid at the cell's location, method 30261 may include setting the new global occupied grid surface confidence to the difference between the old global occupied grid surface confidence and the local occupied grid surface confidence 30359. In 30463, if the new global occupied grid surface confidence is less than zero, method 30261 may include setting the new global occupied grid surface classification to the value of the local occupied grid surface classification 30469.

Referring here to Figure 3M, updating the global occupancy grid using values from a repository of static values 30263 can include, but is not limited to, the following: if, in 30361, there are further cells to be processed within the local occupancy grid, method 30263 may include accessing the log odds from the local occupancy grid 30363 and updating the log odds in the global occupancy grid using the values from the local occupancy grid at that location 30365. If, in 30367, the maximum certainty that a cell is empty is met, and in 30369, the device is moving within a given lane barrier, and in 30371, the surface is traversable, method 30263 may include updating the probability that a cell is occupied 30373 and returning to continue processing further cells. If, in 30367, the maximum certainty that the cell is empty is not reached, or in 30369, the device is not moving within a given lane, or in 30371, the surface is not traversable in the mode in which the device is currently moving, method 30263 may include returning to consider further cells without updating the log odds. If the device is in standard mode, i.e., a mode in which the device can navigate a relatively uniform surface, and the surface classification indicates that the surface is not relatively uniform, method 30263 may adjust the path of the device by updating the log odds (30373) to increase the probability that the cell is occupied. If the device is in standard mode and the surface classification indicates that the surface is relatively uniform, method 30263 may adjust the path of the device by updating the log odds (30373) to decrease the probability that the cell is occupied. When this device is operating in four-wheel mode, i.e., in a mode that allows it to navigate uneven terrain, adjusting the probability of cell occupancy may not be necessary.

Referring here to Figure 4A, the AV can proceed in a specific mode that can be associated with device configurations, for example, the configuration depicted in device 42114A and the configuration depicted in device 42114B. A system of this teaching for real-time control of the device configuration based on at least one environmental factor and the status of the device may include, but is not limited to, a sensor, a moving means, a chassis operably coupled to the sensor and the moving means, the moving means being driven by a motor and a power source, a device processor receiving data from the sensor, and a power base processor controlling the moving means. In some configurations, the device processor can receive environmental data, determine environmental factors, determine configuration changes according to the environmental factors and the status of the device, and provide the configuration changes to the power base processor. The power base processor can issue commands to the moving means to move the device from place to place and physically reconfigure the device when required by the road surface type.

Continuing with Figure 4A, sensors that collect environmental data may include, but are not limited to, cameras, LiDAR, radar, thermometers, pressure sensors, and weather condition sensors, some of which are described herein. From this set of data, the device processor can determine the environmental factors on which device configuration changes may be based. In some configurations, environmental factors may include, but are not limited to, surface factors such as surface type, surface features, and surface conditions. Based on the environmental factors and the current status of the device, the device processor can determine in real time how to modify the configuration to adapt to crossing the detected surface type.

Continuing to refer to Figure 4A, in some configurations, configuration changes of device 42114A/B/C (collectively referred to as device 42114) may include, for example, configuration changes of the moving mechanism. Other configuration changes, such as user information displays and sensor control devices, which may depend on the current mode and surface type, are also conceivable. In some configurations, the moving mechanism may include at least four drive wheels 442101, two on each side of the chassis 42112, and at least two caster wheels 42103 operably coupled to the chassis 42112, as described herein. In some configurations, the drive wheels 442101 may be operably coupled in pairs 42105, each pair 42105 may include a first drive wheel 42101A and a second drive wheel 42101B of the four drive wheels 442101, with each pair 42105 located on opposing sides of the chassis 42112. The operable coupling may include a wheel cluster assembly 42110. In some configurations, a power base processor 41016 (Figure 4B) can control the rotation of the cluster assembly 42110. Left and right wheel motors 41017 (Figure 4B) can drive the wheels 442101 on either side of the chassis 42112. Direction changes can be performed by driving the left and right wheel motors 41017 (Figure 4B) at different rates. A cluster motor 41019 (Figure 4B) can rotate the wheelbase in the forward/rear direction. Wheelbase rotation can allow the cargo to rotate independently of the drive wheels 442101, if so, while the front drive wheel 442101A is higher or lower than the rear drive wheel 442101B, for example, when encountering a discontinuous surface feature. The cluster assembly 42110 allows each pair of wheels 42105 to operate independently, thereby providing forward, reverse, and rotational motion of the device 42114 in response to commands. The cluster assembly 42110 can provide structural support for the pair 42105. The cluster assembly 42110 provides mechanical power to rotate the wheel drive assembly together, enabling functions that depend on the rotation of the cluster assembly, such as, but not limited to, climbing discontinuous surface features, various surface types, and uneven terrain. Further details regarding the operation of the clustered wheels can be found in U.S. Patent Application No. 16,035,205 (Patent Attorney's Reference No. X80), filed July 13, 2018, titled "Mobility Device" (incorporated herein in whole by reference).

Continuing to refer to Figure 4A, the configuration of device 42114 can be associated with, but is not limited to, mode 41033 (Figure 4B) of device 42114. Device 42114 can operate in some of the modes 41033 (Figure 4B). In standard mode 10100-1 (Figure 5E), device 42114B can operate on two of the drive wheels 442101B and two of the caster wheels 42103. Standard mode 10100-1 (Figure 5E) can provide turning and mobility on relatively firm horizontal surfaces, such as, but is not limited to, indoor environments, sidewalks, and pavements. In reinforced mode 10100-2 (Figure 5E) or four-wheel mode, the device 42114A/C can command four of the drive wheels 442101A/B to be actively stabilized via onboard sensors, raising the chassis 42112, casters 42103, and cargo, and/or reorienting them. Four-wheel mode 10100-2 (Figure 5E) provides mobility in various environments and allows the device 42114A/C to climb steep inclines and travel across soft, uneven terrain. In four-wheel mode 10100-2 (Figure 5E), all four of the drive wheels 442101A/B can be deployed, and the caster wheels 42103 can be retracted. Rotation of the cluster 42110 can enable operation on uneven terrain, and the drive wheels 442101A/B can travel over and across discontinuous surface features. This functionality can provide device 42114A/C with mobility in a wide variety of outdoor environments. Device 42114B can operate on solid and stable but wet outdoor surfaces. Frost heave and other natural phenomena can degrade outdoor surfaces, causing cracks and loose material. In four-wheel mode 10100-2 (Figure 5E), device 42114A/C can operate on these degraded surfaces. Mode 41033 (Figure 4B) is described in detail in U.S. Patent No. 6,571,892 (No. '892), issued on June 3, 2003, titled "Control System and Method" (incorporated herein in whole by reference).

Referring here to Figure 4B, the system 41000 can drive device 42114 (Figure 4A) by processing input from sensor 41031, generating commands to wheel motor 41017 to drive wheel 442101 (Figure 4A), and generating commands to cluster motor 41019 to drive cluster 42110 (Figure 4A). The system 41000 may include, but is not limited to, a device processor 41014 and a power base processor 41016. The device processor 41014 may receive and process environmental data 41022 from sensor 41031 and provide configuration information 40125 to the power base processor 41016. In some configurations, the device processor 41014 may include a sensor processor 41021 that can receive and process environmental data 41022 from sensor 41031. Sensor 41031 may include, but is not limited to, a camera as described herein. From this data, information about the travel surface traversed by device 42114 (Figure 4A) can be stored and processed. In some configurations, the travel surface information can be processed in real time. The device processor 41014 may include a configuration processor 41023 that can determine the surface type 40121 from environmental data 41022 traversed by device 42114 (Figure 4A). The configuration processor 41023 may include a travel surface processor 41029 (Figure 4C) that can create a travel surface classification layer, a travel surface confidence layer, and an occupation layer from the environmental data 41022. This data can be used by the power base processor 41016 to create travel commands 40127 and motor commands 40128, as described herein, and by the global occupation grid processor 41025 to update the occupation grid that can be used for route planning. Configuration 40125 can be at least partially based on the surface type 40121. Surface type 40121 and mode 41033 can be used to determine occupied grid information 41022, which may at least partially include the probability that a cell in the occupied grid is occupied. The occupied grid can at least partially enable the determination of possible paths for device 42114 (Figure 4A).

Continuing to refer to Figure 4B, the power base processor 41016 receives configuration information 40125 from the device processor 41014 and can process the configuration information 40125 together with other information, such as routing information. The power base processor 41016 may include a control processor 40325 that can create a move command 40127 based on at least the configuration information 40125 and provide the move command 40127 to the motor drive processor 40326. The motor drive processor 40326 can generate a motor command 40128 that can instruct and move the device 42114 (Figure 4A). Specifically, the motor drive processor 40326 can generate a motor command 40128 that can drive the wheel motor 41017, and a motor command 40128 that can drive the cluster motor 41019.

Referring here to Figure 4C, the real-time surface detection of this teaching may include a configuration processor 41023, which may include, but is not limited to, a travel surface processor 41029. The travel surface processor 41029 can determine the characteristics of the travel surface on which device 42114 (Figure 4A) is navigating. These characteristics can be used to determine the future configuration of device 42114 (Figure 4A). The travel surface processor 41029 may include, but is not limited to, a neural network processor 40207, data transformations 40215, 40219, and 40239, a layer processor 40241, and an occupied grid processor 40242. Together, these components can generate information that can instruct changes to the configuration of device 42114 (Figure 4A) and enable modifications to the occupied grid 40244 (Figure 4C) that can inform the path plan for the progress of device 42114 (Figure 4A).

Referring here to Figures 4C and 4D, the neural network processor 40207 can expose environmental data 41022 (Figure 4B) to a trained neural network that can indicate the type of surface that each point of data collected by the sensor 41031 (Figure 4B) is likely to represent. The environmental data 41022 (Figure 4B) can, but is not limited to, be received as a camera image 40202, and the camera can be associated with camera properties 40204. The camera image 40202 may include a 2D grid of points 40201 having an X resolution 40205 (Figure 4D) and a Y resolution 40204 (Figure 4D). In some configurations, the camera image 40202 may include an RGB-D image, the X resolution 40205 (Figure 4D) may include 40,640 pixels, and the Y resolution 40204 (Figure 4D) may include 40,480 pixels. In some configurations, the camera image 40202 can be converted into an image formatted according to the requirements of the selected neural network. In some configurations, the data can be normalized, scaled, and converted from 2D to 1D, which can improve the processing efficiency of the neural network. The neural network can be trained in many ways, including, but not limited to, training with RGB-D camera images. In some configurations, the trained neural network, represented in the neural network file 40209 (Figure 4D), can be made available to the neural network processor 40207 through a direct connection to the processor running the trained neural network, or, for example, through a communication channel. In some configurations, the neural network processor 40207 can use the trained neural network file 40209 (Figure 4D) to identify surface types 40121 (Figure 4C) in the environmental data 41022 (Figure 4B). In some configurations, surface type 40121 (Figure 4C) can include, but are not limited to, impassable, hard drivable, soft drivable, and curb. In some configurations, surface type 40121 (Figure 4C) can include, but are not limited to, impassable/background, asphalt, concrete, brick, compacted soil, wooden planks, gravel/pebbles, grass, mulch, sand, curb, solid metal, metal grid, tactile paving, snow/ice, and railway tracks. The result of the neural network processing can include a surface classification grid 40303 of points 40213 having an X resolution 40205, a Y resolution 40203, and a center 40211. Each point 40213 within the surface classification grid 40303 can be associated with a likelihood of being one of the specific surface types 40121 (Figure 4C).

Continuing with reference to Figures 4C and 4D, the traveling surface processor 41029 (Figure 4C) may include a 2D-to-3D transformation 40215 that can backproject from a 2D surface classification grid 40303 (Figure 4D) in a 2D camera frame to a 3D image cube 40307 (Figure 4D) in 3D real-world coordinates as seen by the camera. The backprojection can restore the 3D properties of the 2D data and can transform a 2D image from an RGB-D camera into a 3D camera frame 40305 (Figure 4C). Each point 40233 (Figure 4D) within the cube 40307 (Figure 4D) can be associated with a likelihood, depth coordinate, and X/Y coordinate, respectively, which are specific to one of the surface types 40121 (Figure 4C). The dimensions of the point cube 40307 (Figure 4D) can be defined according to camera properties 40204 (Figure 4C), such as, for example, focal length x, focal length y, and projection center 40225, although this is not limited to these properties. For example, camera properties 40204 may include the maximum range over which the camera can reliably project. Furthermore, there may be features of device 42114 (Figure 4A) that may interfere with the image 40202. For example, the caster 42103 (Figure 4A) may interfere with the view of camera 40227 (Figure 4D). These factors can limit the number of points in the point cube 40307 (Figure 4D). In some configurations, camera 40227 (Figure 4D) cannot reliably project beyond about 6 meters, which can represent an upper limit to the range of camera 40227 (Figure 4D) and limit the number of points in the point cube 40307 (Figure 4D). In some configurations, the features of device 42114 (Figure 4A) can act as a minimum limit to the range of camera 40227 (Figure 4D). For example, the presence of caster 42103 (Figure 4A) can imply a minimum limit, which in some configurations may be set to approximately 1 meter. In some configurations, points within point cube 40307 (Figure 4D) can be limited to points 1 meter or more from camera 40227 (Figure 4D), and 6 meters or less from camera 40227 (Figure 4D).

Continuing with reference to Figures 4C and 4D, the traveling surface processor 41029 (Figure 4C) may include a base link transformer 40219 that can transform a 3D cube of points into coordinates associated with the device 42114 (Figure 4A), i.e., a base link frame 40309 (Figure 4C). The base link transformer 40219 can transform a 3D data point 40223 (Figure 4D) in cube 40307 (Figure 4D) into a point 40233 (Figure 4D) in cube 40308 (Figure 4D), where the Z dimension is set to the base of the device 42114 (Figure 4A). The running surface processor 41029 (Figure 4C) may include an OG preparation 40239 that can project point 40233 (Figure 4D) in cube 40308 (Figure 4D) onto the occupied grid 40244 (Figure 4D) as point 40237 (Figure 4D) in cube 40311 (Figure 4D). The layer processor 40241 can flatten point 40237 (Figure 4D) into various layers 40312 (Figure 4C) depending on the data represented by point 40237 (Figure 4D). In some configurations, the layer processor 40241 can apply scalar values to the layers 40312 (Figure 4C). In some configurations, layer 40312 (Figure 4C) may include the probability of the occupied layer 40243, a surface classification layer 40245 (Figure 4D) determined by the neural network processor 40207, and a surface type confidence layer 40247 (Figure 4D). In some configurations, the surface type confidence layer 40247 (Figure 4D) may be determined by converting the class score from the neural network processor 40207 into a score that can be determined by normalizing the class score to a probability distribution across output classes as logarithm(class score)/Σlogarithm(each class). In some configurations, one or more layers may be replaced or extended by layers that provide the probability of unmovable surfaces.

Continuing to refer to Figures 4C and 4D, in some configurations, the probability value in the occupied layer 40243 can be expressed as a log-odds (log-odds -> ln(p/(1-p)) value). In some configurations, the probability value in the occupied layer 40243 can be based on at least a combination of mode 41033 (Figure 4B) and surface type 40121 (Figure 4B). In some configurations, the pre-selected probability value in the occupied layer 40243 is, for example, not limited to However, (1) when surface type 40121 (Figure 4A) is rigid and drivable, and device 42114 (Figure 4A) is in a pre-selected set of mode 41033 (Figure 4B), or (2) when surface type 40121 (Figure 4A) is flexible and drivable, and device 42114 (Figure 4A) is in a specific pre-selected mode such as standard mode, or (3 The configuration can be selected to cover situations such as when chair 42114 (Figure 4A) is in a specific pre-selected mode, such as four-wheel mode, or (4) when surface type 40121 (Figure 4A) is discontinuous and device 42114 (Figure 4A) is in a specific pre-selected mode, such as standard mode, or (5) when surface type 40121 (Figure 4A) is discontinuous and device 42114 (Figure 4A) is in a specific pre-selected mode, such as four-wheel mode, or (6) when surface type 40121 (Figure 4A) is immobile and device 42114 (Figure 4A) is in a pre-selected set of mode 41033 (Figure 4B). In some configurations, the probability values may include, but are not limited to, those listed in Table I. In some configurations, the neural network predicted probabilities may be adjusted as needed and may replace the probabilities listed in Table I.

Referring again to Figure 4C, the occupied grid processor 40242 can provide the global occupied grid processor 41025 in real time with parameters that may affect the probability value of the occupied grid 40244, such as, for example, surface type 40121 and occupied grid information 41022. For example, configuration information 40125 (Figure 4B), such as mode 41033 and surface type 40121, can be provided to the power base processor 41016 (Figure 4B). The power base processor 41016 (Figure 4B) can determine a motor command 40128 (Figure 4B) that can configure the device 42114 (Figure 4A), based at least on the configuration information 40125 (Figure 4B).

Referring here to Figures 4E and 4F, device 42100A can be configured according to this teaching to operate in standard mode. In standard mode, the caster 42103 and the second drive wheel 42101B can be stationary on the ground as device 42100A navigates its path. The first drive wheel 42101A can be raised by a pre-selected amount 42102 (Figure 4F) to pass over the travel surface. Device 42100A can navigate normally on a relatively firm, horizontal surface. When traveling in standard mode, the occupied grid 40244 (Figure 4C) can reflect surface type limits (see Table I), thus allowing for the selection of a suitable mode, or configuration changes based on the surface type and current mode.

Referring here to Figure 4G-4J, the device 42100B/C can be configured according to this teaching to operate in four-wheel mode. In one configuration in four-wheel mode, the first drive wheel 42101A and the second drive wheel 42101B can be stationary on the ground as the device 42100A navigates its path. The caster 42103 can be retracted and can pass over the travel surface by a pre-selected amount 42104 (Figure 4H). In another configuration in four-wheel mode, the first drive wheel 42101A and the second drive wheel 42101B can be substantially stationary on the ground as the device 42100A navigates its path. The caster 42103 can be retracted, and the chassis 42111 can be rotated, for example, to adapt to discontinuous surfaces (thus moving the caster 42103 further away from the ground). In this configuration, the caster 42103 can traverse the travel surface by a pre-selected amount 42108 (Figure 1J). Device 42100A/C can navigate successfully on various surfaces, including soft and discontinuous surfaces. In another configuration in four-wheel mode, the second drive wheel 42101B can be stationary on the ground as device 42100A, while the first drive wheel 42101A can be raised as device 42100C (Figure 1J) navigates its path. The caster 42103 can be retracted, and the chassis 42111 can be rotated, for example, to adapt to discontinuous surfaces (thus moving the caster 42103 further from the ground). When traveling in four-wheel mode, the occupied grid 40244 (Figure 4C) can reflect the surface type (see Table I), thus allowing for the selection of a suitable mode, or configuration changes based on the surface type and current mode.

Referring here to Figure 4K, a method 40150 for real-time control of the device configuration of a device, such as an AV device, progressing along a path based on at least one environmental factor and device configuration, may include, but is not limited to, receiving sensor data 40151, determining the surface type 40153 based at least on the sensor data, and determining the current mode 40155 based at least on the surface type and the current device configuration. Method 40150 may also include determining the next device configuration 40157 based at least on the current mode and surface type, determining a move command 40159 based at least on the next device configuration, and changing the current device configuration to the next device configuration 40161 based at least on the move command.

Referring primarily to Figure 5A, annotated point data 10379 (Figure 5B) can be provided to the device controller 10111 in response to objects appearing in the path of the AV. The annotated point data 10379 (Figure 5B), which may form the basis for route information that can be used to command the AV 10101 (Figure 1A) to proceed along the path, may include, but are not limited to, navigable edges, mapped trajectories such as, but are not limited to, mapped trajectories 10413/10415 (Figure 5D), and labeled features such as, but are not limited to, SDSF 10377 (Figure 5C). The mapped trajectories 10413/10415 (Figure 5C) may include a graph of edges in the route space and initial weights assigned to portions of the route space. The graph of edges may include, but are not limited to, characteristics such as directionality and capacitance, and edges may be categorized according to these characteristics. The mapped trajectories 10413/10415 (Figure 5C) may include cost modifiers associated with the surface of the route space and travel modes associated with the edges. Travel modes may include, but are not limited to, path following and SDSF climbing. Other modes may include, but are not limited to, operation modes such as autonomous, mapping, and wait for intervention. Ultimately, the path can be selected based on at least the lower cost modifier. Forms relatively far from the mapped trajectories 10413/10415 (Figure 5C) may have higher cost modifiers and may not be of much interest when forming the path. The initial weighting is adjusted while AV 10101 (Figure 1A) is operating and may, as a possibility, cause a modification of the path. The adjusted weighting can be used to adjust the edge/weighting graph 10381 (Figure 5B) and may be based on at least the current travel mode, current surface, and edge category.

Continuing with Figure 5A, the device controller 10111 may include a feature processor capable of performing specific tasks related to incorporating the eccentricity of any feature into the path. In some configurations, the feature processor may include, but is not limited to, an SDSF processor 10118. In some configurations, the device controller 10111 may include, but is not limited to, the SDSF processor 10118, the sensor processor 10703, the mode controller 10122, and the base controller 10114, each as described herein. The SDSF processor 10118, the sensor processor 10703, and the mode controller 10122 may provide inputs to the base controller 10114.

Continuing with Figure 5A, the base controller 10114 can determine, based on inputs provided by at least the mode controller 10122, the SDSF processor 10118, and the sensor processor 10703, information that the power base 10112 can use to drive the AV 10101 (Figure 1A) along a path determined by the base controller 10114 based on at least the edge/weighted graph 10381 (Figure 5B). In some configurations, the base controller 10114 can ensure that the AV 10101 (Figure 1A) follows a predetermined path from a starting point to a destination and can modify the predetermined path based on at least external and/or internal conditions. In some configurations, external conditions may include, but are not limited to, stop signals, SDSFs, and obstacles within or near the path being traveled by the AV 10101 (Figure 1A). In some configurations, the internal conditions may include, but are not limited to, mode transitions that reflect the response of AV 10101 (Figure 1A) to external conditions. The device controller 10111 can determine commands to send to the power base 10112 based on at least external and internal conditions. The commands may include, but are not limited to, velocity and direction commands that instruct AV 10101 (Figure 1A) to proceed in a commanded direction at a commanded velocity. Other commands may include, for example, a set of commands that enable feature responses such as SDSF climbing. The base controller 10114 may determine the desired velocity between waypoints in the path by conventional methods, for example, including the Interior Point Optimizer (IPOPT) Large-Scale Nonlinear Optimization (https://projects.coin-or.org/Ipopt). The base controller 10114 can determine a desired path based on, for example, but not limited to, conventional techniques such as Dijkstra's algorithm, A ^* search algorithm, or techniques based on breadth-first search algorithms. The base controller 10114 can form a box around the mapped trajectory 10413/10415 (Figure 5C) to set an area where obstacle detection can be performed. The height of the payload carrier can be adjusted, at least in part, based on a specified velocity, when adjustable.

Continuing to refer to Figure 5A, the base controller 10114 can translate speed and direction determinations into motor commands. For example, but not limited to, when encountering an SDSF such as a curb or slope, the base controller 10114 can instruct the power base 10112 to raise the payload carrier 10173 (Figure 1A), align the AV 10101 (Figure 1A) with the SDSF at an angle of approximately 90°, and reduce the speed to a relatively low level in the SDSF climbing mode. When the AV 10101 (Figure 1A) is climbing a substantially discontinuous surface, the base controller 10114 can instruct the power base 10112 to transition to an climbing phase in which the speed is increased because an increased torque is required to move the AV 10101 (Figure 1A) up the slope. When AV10101 (Figure 1A) encounters a relatively horizontal surface, the base controller 10114 can reduce speed to stay over any flat portion of the SDSF. In the case of a downhill ramp associated with a flat portion, when AV10101 (Figure 1A) begins descending a substantially discontinuous surface, and both wheels are on the downhill ramp, the base controller 10114 can allow speed to increase. For example, when encountering an SDSF such as a slope, the slope can be identified and treated as a structure. The features of the structure may include, for example, a ramp of a pre-selected size. The ramp may include an incline of about 30° and may optionally be on both sides of a flat area. The device controller 10111 (Figure 5A) can distinguish between obstacles and slopes by comparing the angle of the perceived feature with the expected slope/incline angle, and the angle can be received from the sensor processor 10703 (Figure 5A).

Here, primarily referring to Figure 5B, the SDSF processor 10118 can locate navigable edges from blocks of traversable surfaces formed by polygonal meshes represented in annotated point data 10379, which can be used to create paths for crossing by AV 10101 (Figure 1A). Within an SDSF buffer 10407 (Figure 5C), which can form an area of a pre-selected size around an SDSF line 10377 (Figure 5C), the navigable edges can be erased in preparation for special handling in anticipation of an SDSF crossing (see Figure 5D). Closed line segments, such as segment 10409 (Figure 5C), can be drawn to bisect the SDSF buffer 10407 (Figure 5C) between pairs of previously determined SDSF points 10789 (Figure 1N). In some configurations, since a closed line segment is considered a candidate for SDSF crossing, the segment end 10411 (Figure 5C) can be located in an unobstructed portion of the traversable surface, and there can be sufficient room for AV 10101 (Figure 1A) to travel along the line segment between adjacent SDSF points 10789 (Figure 1N), and the area between SDSF points 10789 (Figure 1N) can be the traversable surface. The segment end 10411 (Figure 5C) can be connected to the underlying configuration to form a vertex and a traversable edge. For example, line segments 10461, 10463, 10465, and 10467 (Figure 5C) that satisfy the crossing criterion are shown as part of the configuration in Figure 5D. In contrast, line segment 10409 (Figure 5C) did not satisfy the criterion because, at least, the segment end 10411 (Figure 5C) does not lie on the traversable surface. Overlapping SDSF buffers 10506 (Figure 5C) can indicate SDSF discontinuities, which can disadvantage SDSF crossings of SDSFs within the overlapping SDSF buffers 10506 (Figure 5C). SDSF lines 10377 (Figure 5C) can be smoothed, and the locations of SDSF points 10789 (Figure 1N) can be adjusted so that they are located at a pre-selected distance, based on the area occupied by at least AV 10101 (Figure 1A).

Continuing to refer to Figure 5B, the SDSF processor 10118 can convert the annotated point data 10379 into an edge/weighted graph 10381, including morphological modifications for SDSF cross-sections. The SDSF processor 10118 may include a seventh processor 10601, an eighth processor 10702, a ninth processor 10603, and a tenth processor 10605. The seventh processor 10601 can convert the coordinates of points in the annotated point data 10379 into a global coordinate system, achieve compatibility with GPS coordinates, and generate a GPS-compatible dataset 10602. The seventh processor 10601 can generate the GPS-compatible dataset 10602 using conventional processes such as, for example, affine matrix transformations and PostGIS transformations, but is not limited to these. The World Geodetic System (WGS) can be used as the standard coordinate system because it takes into account the curvature of the Earth. The map can be stored in the Universal Transverse Mercator (UTM) coordinate system and can be switched to the WGS when it is necessary to find the location of a specific address.

Referring primarily to Figure 5C, the eighth processor 10702 (Figure 5B) smooths the SDSF, determines the boundary of the SDSF 10377, and creates a buffer 10407 around the SDSF boundary, increasing the surface cost modifier as it moves further from the SDSF boundary. The mapped trajectories 10413/10415 may be special-case lanes with the lowest cost modifiers. Lower cost modifiers 10406 can generally be located near the SDSF boundary, while higher cost modifiers 10408 can generally be located relatively far from the SDSF boundary. The eighth processor 10702 can provide point cloud data 10704 (Figure 5B) with costs to the ninth processor 10603 (Figure 5B).

Continuing, primarily with reference to Figure 5C, the ninth processor 10603 (Figure 5B) can calculate an approximately 90° approach 10604 (Figure 5B) for AV 10101 (Figure 1A) to traverse an SDSF 10377 that meets the criteria for labeling them as traversable. The criteria may include the SDSF width and the SDSF smoothness. Line segments such as line segment 10409 can be created such that their lengths indicate the minimum approach distance that AV 10101 (Figure 1A) may be required to approach the SDSF 10377 and the minimum exit distance that may be required to exit the SDSF 10377. Segment endpoints such as endpoint 10411 may be integrated with the underlying routing configuration. The criteria used to determine whether an SDSF approach is possible may exclude several possible approaches. SDSF buffers such as SDSF buffer 10407 can be used to calculate effective approach and root morphological margin formation.

Referring again, primarily to Figure 5B, the tenth processor 10605 can generate from a morphology an edge/weight graph 10381, which is an edge and weight graph developed herein, that can be used to calculate a path through the map. The morphology may include cost modifiers and travel modes, and the edges may include directionality and capacity. The weights can be adjusted at runtime based on information from any number of sources. The tenth processor 10605 can provide the base controller 10114 with at least one sequence of ordered points, in addition to a recommended travel mode at a particular point, to enable path generation. Each point in each sequence of points represents the location and labeling of a possible path point on the processed traversable surface. In some configurations, the labeling may indicate that the point represents a part of the features that may be encountered along the path, such as, for example, an SDSF, etc. In some configurations, the features may be further labeled using suggested processing based on the type of feature. For example, in some configurations, if a path point is labeled as an SDSF, further labeling may include a mode. This mode can be interpreted by AV10101 (Figure 1A) as a suggested travel command for AV10101 (Figure 1A), such as switching AV10101 (Figure 1A) to SDSF uphill mode 100-31 (Figure 5E) to allow AV10101 (Figure 1A) to traverse SDSF 10377 (Figure 5C).

Referring here to Figure 5E, in some configurations, the mode controller 10122 can provide instructions to the base controller 10114 (Figure 5A) to perform mode transitions. The mode controller 10122 can establish the mode in which the AV 10101 (Figure 1A) is progressing. For example, the mode controller 10122 can provide the base controller 10114 with a change in mode indication, for example, between route-following mode 10100-32 and SDSF climbing mode 10100-31 when the SDSF is identified along the progress path. In some configurations, annotated point data 10379 (Figure 5B) can include mode identifiers at various points along the route, for example, when the mode is changed to adapt the route. For example, if the SDSF 10377 (Figure 5C) is labeled within the annotated point data 10379 (Figure 5B), the device controller 10111 can determine the mode identifier associated with the route point and, potentially, adjust the commands to the power base 10112 (Figure 5A) based on the desired mode. In addition to the SDSF climbing mode 10100-31 and path-following mode 10100-32, in some configurations, the AV 10101 (Figure 1A) can support operating modes, which may include, but are not limited to, the standard mode 10100-1 and the enhanced (four-wheel) mode 10100-2 described herein. The height of the payload carrier 10173 (Figure 1A) can be adjusted to provide the necessary clearance across obstacles and along slopes.

Referring here to Figure 5F, a method 11150 for navigating an AV towards a target point that traverses at least one SDSF may include, but is not limited to, receiving SDSF information related to the SDSF, the location of the target point, and the location of the AV. The SDSF information may, but is not limited to, a set of points classified as SDSF points and the associated probability for each point that a point is an SDSF point. Method 11150 may include drawing a closed polygon encompassing the location of the AV and the location of the target point, and drawing a path line between the target point and the location of the AV. The closed polygon may include a pre-selected width. Table I contains a range of possible pre-selected variables discussed herein. Method 11150 may include selecting two of the SDSF points located within the polygon, 11155, and drawing an SDSF line between the two points. In some configurations, the selection of SDSF points may be random or by any other method. If, in step 11159, fewer than a first pre-selected number of points exist within a first pre-selected distance of the SDSF line, and in step 11161, fewer than a second pre-selected number of attempts exist in selecting SDSF points, drawing lines between them and having fewer than a first pre-selected number of points around the SDSF line, then method 11150 may include returning to step 11155. If, in step 11161, a second pre-selected number of attempts exist in selecting SDSF points, drawing lines between them and having fewer than a first pre-selected number of points around the SDSF line, then method 11150 may include acknowledging that no SDSF lines were detected.

Here, primarily referring to Figure 5G, if in 11159 (Figure 5F) there are a first pre-selected number of points or more, method 11150 may include fitting the curve to the points that fall within a first pre-selected distance of the SDSF line 11165. If in 11167 the number of points within the first pre-selected distance of the curve exceeds the number of points within the first pre-selected distance of the SDSF line, and in 11171 the curve intersects a path line, and in 11173 there are no gaps between points on the curve beyond a second pre-selected distance, method 11150 may include identifying the curve as an SDSF line 11175. If, in step 11167, the number of points within the first pre-selected distance of the curve does not exceed the number of points within the first pre-selected distance of the SDSF line, or in step 11171, the curve does not intersect the path line, or in step 11173, there are gaps between points on the curve that exceed the second pre-selected distance, and in step 11177, the SDSF line does not remain stable, and in step 11169, the curve fit has not been attempted more than the second pre-selected number of times, then method 11150 may return to step 11165. A stable SDSF line is the result of subsequent iterations that yield the same or fewer points.

Here, primarily referring to Figure 5H, if, in 11169 (Figure 5G), the curve fit has been attempted only a second pre-selected number of times, or if, in 11177 (Figure 5G), the SDSF line remains stable or degrades, method 11150 may include receiving occupancy grid information 11179. The occupancy grid can provide the probability of an obstacle being present at a point. The occupancy grid information can enhance the SDSF and path information found within the polygon surrounding the AV path and the SDSF, when the occupancy grid includes data captured and/or calculated over a common geographical area with the polygon. Method 11150 may include selecting a point from the common geographical area and its associated probability 11181. In step 11183, if the probability that an obstacle is present at the selected point is higher than a pre-selected percentage, and in step 11185, if the obstacle is located between AV and the target point, and in step 11186, if the obstacle is located less than a third pre-selected distance from the SDSF line between the SDSF line and the target point, method 11150 may include projecting the obstacle onto the SDSF line 11187. In step 11183, if the probability that a location contains an obstacle is less than or equal to a pre-selected percentage, or in step 11185, if the obstacle is not located between AV and the target point, or in step 11186, if the obstacle is located at a distance equal to or greater than a third pre-selected distance from the SDSF line between the SDSF and the target point, and in step 11189, if there are further obstacles to be dealt with, method 11150 may include resuming the processing in step 11179.

Here, primarily referring to Figure 5I, in 11189 (Figure 5H), if there are no further obstacles to be dealt with, method 11150 may include connecting the projections and finding the endpoints of the connected projections along the SDSF line 11191. Method 11150 may include marking a portion of the SDSF line between the projection endpoints as non-crossable 11193. Method 11150 may include marking a portion of the SDSF line outside the non-crossable section as crossable 11195. Method 11150 may include redirecting the AV to within a fifth pre-selected amount perpendicular to the crossable section of the SDSF line 11197. In 11199, if the azimuth error with respect to a line perpendicular to the crossable section of the SDSF line exceeds a first pre-selected amount, method 11150 may include decelerating the AV by a ninth pre-selected amount 11251. Method 11150 may include moving the AV forward toward the SDSF line and decelerating it by a second pre-selected amount per meter distance between the AV and the traversable SDSF line 11253. If, in 11255, the distance of the AV from the traversable SDSF line is less than a fourth pre-selected distance, and in 11257, the azimuth error is greater than or equal to a third pre-selected amount relative to a line perpendicular to the SDSF line, method 11150 may include decelerating the AV by a ninth pre-selected amount 11252.

Here, primarily referring to Figure 5J, in 11257 (Figure 5I), if the azimuth error is less than a third pre-selected amount relative to a line perpendicular to the SDSF line, method 11150 may include ignoring the updated SDSF information and driving the AV at a pre-selected speed 11260. In 11259, if the rise of the front portion of the AV relative to the rear portion of the AV is between a sixth pre-selected amount and a fifth pre-selected amount, method 11150 may include driving the AV forward and increasing the speed of the AV to an eighth pre-selected amount per degree of rise 11261. In 11263, if the rise of the front portion of the AV relative to the rear portion is less than a sixth pre-selected amount, method 11150 may include driving the AV forward at a seventh pre-selected speed 11265. In step 11267, if the rear of the AV is greater than a fifth pre-selected distance from the SDSF line, method 11150 may include acknowledging that the AV has completed crossing the SDSF 11269. In step 11267, if the rear of the AV is less than, or equal to, a fifth pre-selected distance from the SDSF line, method 11150 may include returning to step 11260.

Referring here to Figure 5K, the system 51100 for navigating the AV towards a target point traversing at least one SDSF may include, but is not limited to, a path line processor 11103, an SDSF detector 11109, and an SDSF controller 11127. The system 51100 can be operably coupled with a surface processor 11601 capable of processing sensor information, which may, for example, include, but is not limited to, images of the vicinity of the AV 10101 (Figure 5L). The surface processor 11601 can provide real-time surface feature updates, including indications of SDSFs. In some configurations, a camera may provide RGB-D data, which may classify its points according to surface type. In some configurations, the system 51100 can process points classified as SDSFs and their associated probabilities. The system 51100 can be operably coupled with a system controller 11602 capable of managing aspects of the operation of the AV 10101 (Figure 5L). The system controller 11602 can maintain an occupied grid 11138 which may include information from available sources regarding the navigable area in the vicinity of AV 10101 (Figure 5L). The occupied grid 11138 may include the probability of obstacles being present. This information, in conjunction with the SDSF information, can be used to determine whether SDSF 10377 (Figure 5N) can be traversed by AV 10101 (Figure 5L) without encountering obstacle 11681 (Figure 5M). Based on the environment and other information, the system controller 11602 can determine a speed limit 11148 that AV 10101 (Figure 5N) should not exceed. The speed limit 11148 can be used as a guideline for the speed set by system 51100, or it can be overridden. System 51100 can be operably coupled to the base controller 10114, which can transmit travel commands 11144 generated by the SDSF controller 11127 to the travel components of AV 10101 (Figure 5L). The base controller 10114 can provide the SDSF controller 11127 with information about the orientation of AV 10101 (Figure 5L) during SDSF crossing.

Continuing to refer to Figure 5K, the path line processor 11103 can continuously receive surface classification points 10789 in real time, which may include, but are not limited to, points classified as SDSF. The path line processor 11103 can receive the location of the target point 11139 and the AV location 11141, which may be, for example, indicated by the center 11202 (Figure 5L) of AV 10101 (Figure 5L). The system 51100 may include a polygon processor 11105 that draws a polygon 11147 encompassing AV location 11141, the location of the target point 11139, and the path 11214 between the target point 11139 and AV location 11141. The polygon 11147 may include a pre-selected width. In some configurations, the pre-selected width may include the approximate width of AV 10101 (Figure 5L). The SDSF point 10789 located within polygon 11147 can be identified.

Continuing to refer to Figure 5K, the SDSF detector 11109 can receive surface classification points 10789, paths 11214, polygons 11147, and target points 11139, and can determine the most suitable SDSF line 10377 according to the criteria described herein, which are available in the incoming data. The SDSF detector 11109 may include, but is not limited to, a point processor 11111 and an SDSF line processor 11113. The point processor 11111 may include selecting two of the SDSF points 10789 located within polygon 11147 and drawing an SDSF line 10377 between the two points. If there are fewer than a first pre-selected number of points within a first pre-selected distance of the SDSF line 10377, and if there are fewer than a second pre-selected number of attempts in selecting an SDSF point 10789, and a line is drawn between the two points, with fewer than a first pre-selected number of points around the SDSF line, the point processor 11111 may again loop through the selection-draw-test loop as described herein. If there are a second pre-selected number of attempts in selecting SDSF points, and a line is drawn between them, with fewer than a first pre-selected number of points around the SDSF line, the point processor 11111 may recognize that no SDSF line was detected.

Continuing to refer to Figure 5K, the SDSF line processor 11113 may, if there are a first pre-selected number or more of points 10789, fit the curve 11609-11611 (Figure 5L) to the points 10789 that fall within the first pre-selected distance of the SDSF line 10377. The SDSF line processor 11113 may identify curve 11609-11611 (Figure 5L) as (for example) SDSF line 10377 if the number of points 10789 within a first pre-selected distance of curve 11609-11611 (Figure 5L) exceeds the number of points 10789 within a first pre-selected distance of SDSF line 10377, and if curve 11609-11611 (Figure 5L) intersects with path line 11214, and if there are no gaps between points 10789 on curve 11609-11611 (Figure 5L) beyond a second pre-selected distance. If the number of points 10789 within a pre-selected distance of curve 11609–11611 (Figure 5L) does not exceed the number of points 10789 within a first pre-selected distance of SDSF line 10377, or if curve 11609–11611 (Figure 5L) does not intersect with path line 11214, or if there are gaps between points 10789 on curve 11609–11611 (Figure 5L) that exceed a second pre-selected distance, and if SDSF line 10377 is not stable, and if the curve fit has not been attempted more than the second pre-selected number of times, the SDSF line processor 11113 can run the curve fit loop again.

Continuing to refer to Figure 5K, the SDSF controller 11127 can receive SDSF lines 10377, occupied grids 11138, AV orientation changes 11142, and speed limits 11148, and can generate SDSF commands 11144 to cause AV 10101 (Figure 5L) to travel so as to correctly traverse SDSF 10377 (Figure 5N). The SDSF controller 11127 may, but is not limited to, include obstacle processors 11115, SDSF approaches 11131, and SDSF crossings 11133. The obstacle processor 11115 can receive the SDSF line 10377, the target point 11139, and the occupied grid 11138, and can determine from among the obstacles identified in the occupied grid 11138 whether any of them could interfere with AV 10101 (Figure 5N) as it crosses the SDSF 10377 (Figure 5N). The obstacle processor 11115 may include, but is not limited to, an obstacle selector 11117, an obstacle tester 11119, and a crossing locator 11121. The obstacle selector 11117 may include, but is not limited to, receiving the occupied grid 11138 as described herein. The obstacle selector 11117 may select occupied grid points and their associated probabilities from a geographic area common to both the occupied grid 11138 and the polygon 11147. If the probability of an obstacle being present at a selected grid point is higher than a pre-selected percentage, and the obstacle is located between AV 10101 (Figure 5L) and target point 11139, and the obstacle is less than a third pre-selected distance from SDSF line 10377 between SDSF line 10377 and target point 11139, the obstacle tester 11119 may project the obstacle onto SDSF line 10377 to form a projection 11621 intersecting SDSF line 10377. If the probability of a location containing an obstacle is less than or equal to a pre-selected percentage, or if the obstacle is not located between AV 10101 (Figure 5L) and target point 11139, or if the obstacle is located at a distance equal to or greater than a third pre-selected distance from SDSF line 10377 between SDSF line 10377 and target point 11139, the obstacle tester 11119 may include resuming execution in receiving the occupied grid 11138 if further obstacles to be processed exist.

Continuing with Figure 5K, the transverse locator 11121 may include connecting projection points and locating the endpoints 11622/11623 (Figure 5M) of the connected projection 11621 (Figure 5M) along the SDSF line 10377. The transverse locator 11121 may also include marking the portion 11624 (Figure 5M) of the SDSF line 10377 between the projection endpoints 11622/11623 (Figure 5M) as non-traversable. The transverse locator 11121 may also include marking the portion 11626 (Figure 5M) of the SDSF line 10377 outside the non-traversable portion 11624 (Figure 5M) as traversable.

Continuing to refer to Figure 5K, the SDSF approach 11131 may include sending an SDSF command 11144 to redirect the AV 10101 (Figure 5N) to within a fifth pre-selected amount perpendicular to the traversable portion 11626 (Figure 5N) of the SDSF line 10377. If the azimuth error with respect to the vertical line 11627 (Figure 5N), which is perpendicular to the traversable portion 11626 (Figure 5N) of the SDSF line 10377, exceeds a first pre-selected amount, the SDSF approach 11131 may include sending an SDSF command 11144 to decelerate the AV 10101 (Figure 5N) by a ninth pre-selected amount. In some configurations, the ninth pre-selected amount can range from very slow to a complete stop. The SDSF approach 11131 may include sending an SDSF command 11144 to move AV 10101 (Figure 5N) forward toward SDSF line 10377, and sending an SDSF command 11144 to decelerate AV 10101 (Figure 5N) by a second pre-selected amount per meter traveled. If the distance between AV 10101 (Figure 5N) and the traversable SDSF line 11626 (Figure 5N) is less than a fourth pre-selected distance, and the azimuth error is greater than or equal to a third pre-selected amount relative to a line perpendicular to SDSF line 10377, the SDSF approach 11131 may also include sending an SDSF command 11144 to decelerate AV 10101 (Figure 5N) by a ninth pre-selected amount.

Continuing to refer to Figure 5K, if the azimuth error is less than a third pre-selected amount relative to a line perpendicular to the SDSF line 10377, the SDSF crossing 11133 may include sending an SDSF command 11144 to ignore the updated SDSF information and drive AV 10101 (Figure 5N) at a pre-selected rate. If the AV orientation change 11142 indicates that the rise of the leading edge 11701 (Figure 5N) of AV 10101 (Figure 5N) relative to the trailing edge 11703 (Figure 5N) of AV 10101 (Figure 5N) is between a sixth pre-selected amount and a fifth pre-selected amount, the SDSF crossing 11133 may include sending an SDSF command 11144 to move AV 10101 (Figure 5N) forward and an SDSF command 11144 to increase the speed of AV 10101 (Figure 5N) to a pre-selected rate per degree of rise. If the AV orientation change 11142 indicates that the rise of the leading edge 11701 (Figure 5N) relative to the trailing edge 11703 (Figure 5N) of AV 10101 (Figure 5N) is less than a sixth pre-selected amount, the SDSF crossing 11133 may transmit an SDSF command 11144 to cause AV 10101 (Figure 5N) to travel forward at a seventh pre-selected speed. If the AV location 11141 indicates that the trailing edge 11703 (Figure 5N) is beyond a fifth pre-selected distance from the SDSF line 10377, the SDSF crossing 11133 may acknowledge that AV 10101 (Figure 5N) has completed crossing the SDSF 10377. If AV location 11141 indicates that the trailing edge 11703 (Figure 5N) is less than or equal to a fifth pre-selected distance from SDSF line 10377, the SDSF crossing 11133 may include repeating a loop that begins with ignoring the updated SDSF information.

Some illustrative ranges of the pre-selected values described herein may include, but are not limited to, those outlined in Table II.

Referring here to Figure 5O, in some configurations, to support real-time data accumulation, the system of this teaching can generate locations in three-dimensional space of various surface types in response to receiving data such as, for example, RGD-D camera image data. The system can rotate images 12155 and convert them from camera coordinate system 12157 to UTM coordinate system 12159. The system can generate polygonal files from the converted images, and the polygonal files can represent three-dimensional locations associated with surface type 12161. A method 12150 for locating features 12151 from camera images 12155 received by AV 10101, having orientation 12163, may include, for example, receiving camera images 12155 by AV 10101. Each camera image 12155 may include an image timestamp 12171, and each image 12155 may include an image color pixel 12167 and an image depth pixel 12169. Method 12150 may include receiving a posture 12163 of AV 10101, wherein the posture 12163 has a posture timestamp 12171, and determining a selected image 12173 by identifying an image from a camera image 12155 having an image timestamp 12165 adjacent to the posture timestamp 12171. Method 12150 may include separating image color pixels 12167 from image depth pixels 12169 within a selected image 12173, providing the image color pixels 12167 to a first machine learning model 12177, and providing the image depth pixels 12169 to a second machine learning model 12179, thereby determining the image surface classification 12161 for the selected image 12173. Method 12150 may also include determining the perimeter points 12181 of a feature within the camera image 12173, where the feature includes feature pixels 12151 within the perimeter, each feature pixel 12151 having the same surface classification 12161, and each perimeter point 12181 having a set of coordinates 12157. Method 12150 may also include converting each of the coordinate sets 12157 to UTM coordinates 12159.

The structure of this instruction relates to a computer system for carrying out the methods discussed herein and a computer-readable medium containing programs for carrying out these methods. Raw data and results can be stored, printed, displayed, transferred to another computer, and/or transferred to another location for future reading and processing. Communication links can be wired or wireless, for example, using cellular communication systems, military communication systems, and satellite communication systems. Parts of this system can operate on a computer with a variable number of CPUs. Other alternative computer platforms may also be used.

This configuration also covers software for performing the methods discussed herein and computer-readable media for storing the software for performing these methods. The various modules described herein may be performed on the same CPU or on different computers. In accordance with the law, this configuration has been described in more or less specific language with respect to its structural and methodological features. However, it should be understood that this configuration is not limited to the specific features shown and described, as the means disclosed herein constitute a preferred form of embodying this configuration.

The method can be implemented electronically, either entirely or in part. Signals representing actions taken by the System and other disclosed components of the System can be transmitted via at least one live communication network. Control and data information can be electronically executed and stored on at least one computer-readable medium. The System can be implemented to run on at least one computer node in at least one live communication network. General forms of at least one computer-readable medium include, but are not limited to, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic media, compact disk read-only memory or any other optical media, punch cards, paper tapes, or any other physical media with perforation patterns, random access memory, programmable read-only memory, and erasable programmable read-only memory (EPROM), flash EPROM, or any other memory chip or cartridge, or any other medium from which a computer can read. Furthermore, at least one computer-readable medium may contain graphs in any form, provided they are appropriately licensed, including, but are not limited to, Graphics Interchange Format (GIF), Joint Photographic Professional Group (JPEG), Portable Network Graphics (PNG), Scalable Vector Graphics (SVG), and Tagged Image File Format (TIFF).

While these instructions have been described above in terms of specific configurations, it should be understood that they are not limited to these disclosed configurations. Many modifications and other configurations are intended and will be covered by both the disclosure and the accompanying claims, as this would be recalled by those skilled in the art. The scope of these instructions is intended to be determined by the proper interpretation and construction of the accompanying claims and their legal equivalents, as understood by those skilled in the art relying on the disclosures in this specification and the accompanying drawings.

Claims (11)

A method for real-time control of the configuration of a device, the device comprising: a chassis; four wheels ; a first side of the chassis operably coupled to one of the four wheels ; and a second side of the chassis operably coupled to another of the four wheels , wherein the method
Receiving environmental data and,
Based on the aforementioned environmental data, the surface type is determined,
Based on the surface type and the first configuration, the mode is determined,
Based on the mode and the surface type, a second configuration is determined,
Based on the second configuration described above , the movement command is determined,
Controlling the configuration of the device based on the move command, including changing the device from a first configuration to a second configuration,
To determine the move command based on an occupied grid representing the world for the purpose of route planning for the device,
Includes,
The environmental data includes RGB-D image data, road surface morphology data, landmark data, long-range camera data, short-range camera data, LIDAR data, radar data, thermometer data, pressure sensor data, weather condition sensor data, and combinations thereof.
method . The method according to claim 1 , further comprising incorporating the occupied grid based at least on the surface type and the mode. The method according to claim 1, wherein the control of the configuration includes coordinated power supply of a first pair of the four-wheel clusters and a second pair of the four-wheel clusters based on the environmental data. The method according to claim 1, wherein the control of the configuration is to drive the four wheels and pairs of casters configured to be movable in the vertical direction , the pairs of casters being operably coupled to the chassis, and the control is to drive two wheels, including a clustered first pair and a clustered second pair which are rotated to lift the first front wheel and the second front wheel, the device being stationary on the first rear wheel, the second rear wheel and the pairs of casters, the clustered first pair including the first front wheel and the first rear wheel, and the clustered second pair including the second front wheel and the second rear wheel . The method according to claim 3, wherein the control of the configuration includes rotating a pair of clusters operably coupled with two first powered wheels on the first side and two second powered wheels on the second side, based on the environmental data . The method according to claim 1, wherein the device further comprises a cargo container, the cargo container is mounted on the chassis, and the chassis controls the height of the cargo container. The height of the cargo container is determined by the method according to claim 6 , based on the environmental data. A system for real-time control of the configuration of a device, wherein the device includes a chassis, four wheels , a first side of the chassis, and a second side of the chassis, and the system is
A device processor, the device processor is configured to receive environmental data surrounding the device, determine a surface type based on the environmental data , determine a mode based on the surface type and a first configuration, and determine a second configuration based on the mode and the surface type,
A power base processor that determines a move command based on the second configuration, the power base processor controls the configuration of the device based on the move command and changes the device from a first configuration to a second configuration, comprising:
The power base processor determines the move command based on an occupied grid representing the world for the purpose of route planning for the device,
The environmental data includes RGB-D image data, road surface morphology data, landmark data, long-range camera data, short-range camera data, LIDAR data, radar data, thermometer data, pressure sensor data, weather condition sensor data, and combinations thereof.
system . The system according to claim 8, wherein the occupied grid comprises information relating to surface type and/or mode. The system according to claim 8, wherein the control of the configuration includes coordinated power supply of a first pair of the four-wheel clusters and a second pair of the four-wheel clusters based on the environmental data . Control of the configuration is to drive the four wheels and pairs of casters configured to be movable in the vertical direction , the pairs of casters being operably coupled to the chassis, and the system is to drive two wheels, including a clustered first pair and a clustered second pair which are rotated to lift the first front wheels and the second front wheels, the device being stationary on the first rear wheels, the second rear wheels and the pairs of casters, the clustered first pair including the first front wheels and the first rear wheels, and the clustered second pair including the second front wheels and the second rear wheels .