Module enforce_sorting

EnforceSorting optimizer rule inspects the physical plan with respect to local sorting requirements and does the following:

• Adds a SortExec when a requirement is not met,
• Removes an already-existing SortExec if it is possible to prove that this sort is unnecessary

The rule can work on valid and invalid physical plans with respect to sorting requirements, but always produces a valid physical plan in this sense.

A non-realistic but easy to follow example for sort removals: Assume that we somehow get the fragment

SortExec: expr=[nullable_col@0 ASC]
  SortExec: expr=[non_nullable_col@1 ASC]

in the physical plan. The first sort is unnecessary since its result is overwritten by another SortExec. Therefore, this rule removes it from the physical plan.

Modules

replace_with_order_preserving_variants

Optimizer rule that replaces executors that lose ordering with their order-preserving variants when it is helpful; either in terms of performance or to accommodate unbounded streams by fixing the pipeline.

sort_pushdown

Structs

EnforceSorting

This rule inspects SortExec’s in the given physical plan in order to remove unnecessary sorts, and optimize sort performance across the plan.

Functions

ensure_sorting

This function enforces sorting requirements and makes optimizations without violating these requirements whenever possible. Requires a bottom-up traversal.

parallelize_sorts

Transform CoalescePartitionsExec + SortExec into SortExec + SortPreservingMergeExec as illustrated below:

Type Aliases

PlanWithCorrespondingCoalescePartitions

This object is used within the EnforceSorting rule to track the closest CoalescePartitionsExec descendant(s) for every child of a plan. The data attribute stores whether the plan is a CoalescePartitionsExec or is connected to a CoalescePartitionsExec via its children.

PlanWithCorrespondingSort

This context object is used within the EnforceSorting rule to track the closest SortExec descendant(s) for every child of a plan. The data attribute stores whether the plan is a SortExec or is connected to a SortExec via its children.