Struct SchemaMapping

pub struct SchemaMapping { /* private fields */ }

The SchemaMapping struct holds a mapping from the file schema to the table schema and any necessary type conversions.

Note, because map_batch and map_partial_batch functions have different needs, this struct holds two schemas:

1. The projected table schema
2. The full table schema

map_batch is used by the ParquetOpener to produce a RecordBatch which has the projected schema, since that’s the schema which is supposed to come out of the execution of this query. Thus map_batch uses projected_table_schema as it can only operate on the projected fields.

map_partial_batch is used to create a RecordBatch with a schema that can be used for Parquet predicate pushdown, meaning that it may contain fields which are not in the projected schema (as the fields that parquet pushdown filters operate can be completely distinct from the fields that are projected (output) out of the ParquetSource). map_partial_batch thus uses table_schema to create the resulting RecordBatch (as it could be operating on any fields in the schema).

Trait Implementations

impl Debug for SchemaMapping

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl SchemaMapper for SchemaMapping

fn map_batch(&self, batch: RecordBatch) -> Result<RecordBatch>

Adapts a RecordBatch to match the projected_table_schema using the stored mapping and conversions. The produced RecordBatch has a schema that contains only the projected columns, so if one needs a RecordBatch with a schema that references columns which are not in the projected, it would be better to use map_partial_batch

fn map_partial_batch(&self, batch: RecordBatch) -> Result<RecordBatch>

Adapts a RecordBatch’s schema into one that has all the correct output types and only contains the fields that exist in both the file schema and table schema.

Unlike map_batch this method also preserves the columns that may not appear in the final output (projected_table_schema) but may appear in push down predicates