Module datafusion::physical_plan

re-export of datafusion_physical_plan crate

Modules

• aggregates
  Aggregates functionalities
• analyze
  Defines the ANALYZE operator
• coalesce_batches
  CoalesceBatchesExec combines small batches into larger batches for more efficient use of vectorized processing by upstream operators.
• coalesce_partitions
  Defines the merge plan for executing partitions in parallel and then merging the results into a single partition
• common
  Defines common code used in execution plans
• display
  Implementation of physical plan display. See crate::displayable for examples of how to format
• empty
  EmptyRelation execution plan
• explain
  Defines the EXPLAIN operator
• expressions
  Defines physical expressions that can evaluated at runtime during query execution
• filter
  FilterExec evaluates a boolean predicate against all input batches to determine which rows to include in its output batches.
• functions
  Declaration of built-in (scalar) functions. This module contains built-in functions’ enumeration and metadata.
• hash_utils
  Functionality used both on logical and physical plans
• insert
  Execution plan for writing data to DataSinks
• joins
  DataFusion Join implementations
• limit
  Defines the LIMIT plan
• memory
  Execution plan for reading in-memory batches of data
• metrics
  Metrics for recording information about execution
• projection
  Defines the projection execution plan. A projection determines which columns or expressions are returned from a query. The SQL statement SELECT a, b, a+b FROM t1 is an example of a projection on table t1 where the expressions a, b, and a+b are the projection expressions. SELECT without FROM will only evaluate expressions.
• repartition
  This file implements the RepartitionExec operator, which maps N input partitions to M output partitions based on a partitioning scheme, optionally maintaining the order of the input rows in the output.
• sorts
  Sort functionalities
• stream
  Stream wrappers for physical operators
• streaming
  Generic plans for deferred execution: StreamingTableExec and PartitionStream
• tree_node
  This module provides common traits for visiting or rewriting tree nodes easily.
• udaf
  This module contains functions and structs supporting user-defined aggregate functions.
• udf
  UDF support
• union
  The Union operator combines multiple inputs with the same schema
• unnest
  Defines the unnest column plan for unnesting values in a column that contains a list type, conceptually is like joining each row with all the values in the list column.
• values
  Values execution plan
• windows
  Physical expressions for window functions

Macros

• internal_err
  Macro wraps Err($ERR) to add backtrace feature

Structs

• ColumnStatistics
  Statistics for a column within a relation
• DefaultDisplay
  A newtype wrapper to display T implementingDisplayAs using the Default mode
• EmptyRecordBatchStream
  EmptyRecordBatchStream can be used to create a RecordBatchStream that will produce no results
• Metric
  Something that tracks a value of interest (metric) of a DataFusion ExecutionPlan execution.
• Statistics
  Statistics for a relation Fields are optional and can be inexact because the sources sometimes provide approximate estimates for performance reasons and the transformations output are not always predictable.
• TopK
  Global TopK
• VerboseDisplay
  A newtype wrapper to display T implementing DisplayAs using the Verbose mode

Enums

• ColumnarValue
  Represents the result of evaluating an expression: either a single ScalarValue or an ArrayRef.
• DisplayFormatType
  Options for controlling how each ExecutionPlan should format itself
• Distribution
  How data is distributed amongst partitions. See Partitioning for more details.
• Partitioning
  Output partitioning supported by ExecutionPlans.

Traits

• Accumulator
  Describes an aggregate functions’s state.
• AggregateExpr
  An aggregate expression that:
• DisplayAs
  Trait for types which could have additional details when formatted in Verbose mode
• ExecutionPlan
  Represent nodes in the DataFusion Physical Plan.
• ExecutionPlanVisitor
  Trait that implements the Visitor pattern for a depth first walk of ExecutionPlan nodes. pre_visit is called before any children are visited, and then post_visit is called after all children have been visited.
• PhysicalExpr
  Expression that can be evaluated against a RecordBatch A Physical expression knows its type, nullability and how to evaluate itself.
• RecordBatchStream
  Trait for types that stream arrow::record_batch::RecordBatch
• WindowExpr
  Common trait for window function implementations

Functions

• accept
  Visit all children of this plan, according to the order defined on ExecutionPlanVisitor.
• collect
  Execute the ExecutionPlan and collect the results in memory
• collect_partitioned
  Execute the ExecutionPlan and collect the results in memory
• displayable
  Return a wrapper around an ExecutionPlan which can be displayed in various easier to understand ways.
• execute_stream
  Execute the ExecutionPlan and return a single stream of results
• execute_stream_partitioned
  Execute the ExecutionPlan and return a vec with one stream per output partition
• need_data_exchange
  Indicate whether a data exchange is needed for the input of plan, which will be very helpful especially for the distributed engine to judge whether need to deal with shuffling. Currently there are 3 kinds of execution plan which needs data exchange 1. RepartitionExec for changing the partition number between two ExecutionPlans 2. CoalescePartitionsExec for collapsing all of the partitions into one without ordering guarantee 3. SortPreservingMergeExec for collapsing all of the sorted partitions into one with ordering guarantee
• project_schema
  Applies an optional projection to a SchemaRef, returning the projected schema
• unbounded_output
• visit_execution_plan
  Recursively calls pre_visit and post_visit for this node and all of its children, as described on ExecutionPlanVisitor
• with_new_children_if_necessary
  Returns a copy of this plan if we change any child according to the pointer comparison. The size of children must be equal to the size of ExecutionPlan::children().

Type Aliases

• SendableRecordBatchStream
  Trait for a [Stream] of RecordBatches