Module datasource

DataFusion data sources: TableProvider and ListingTable

Re-exports

pub use self::default_table_source::provider_as_source;

pub use self::default_table_source::source_as_provider;

pub use self::default_table_source::DefaultTableSource;

pub use self::memory::MemTable;

pub use self::view::ViewTable;

Modules

avro_to_arrow

This module contains code for reading Avro data into RecordBatches

cte_worktable

CteWorkTable implementation used for recursive queries

default_table_source

Default TableSource implementation used in DataFusion physical plans

dynamic_file

dynamic_file_schema contains an UrlTableFactory implementation that can create a ListingTable from the given url.

empty

EmptyTable useful for testing.

file_format

Module containing helper methods for the various file formats See write.rs for write related helper methods

listing

A table that uses the ObjectStore listing capability to get the list of files to process.

listing_table_factory

Factory for creating ListingTables with default options

memory

MemTable for querying Vec<RecordBatch> by DataFusion.

object_store

ObjectStoreRegistry holds all the object stores at Runtime with a scheme for each store. This allows the user to extend DataFusion with different storage systems such as S3 or HDFS and query data inside these systems.

physical_plan

Execution plans that read file formats

provider

Data source traits

schema_adapter

SchemaAdapter and SchemaAdapterFactory to adapt file-level record batches to a table schema.

source

DataSource and DataSourceExec

stream

TableProvider for stream sources, such as FIFO files

view

View data source which uses a LogicalPlan as it’s input.

Enums

TableType

Indicates the type of this table for metadata/catalog purposes.

Traits

TableProvider

A table which can be queried and modified.

Functions

get_statistics_with_limit

Get all files as well as the file level summary statistics (no statistic for partition columns). If the optional limit is provided, includes only sufficient files. Needed to read up to limit number of rows. collect_stats is passed down from the configuration parameter on ListingTable. If it is false we only construct bare statistics and skip a potentially expensive call to multiunzip for constructing file level summary statistics.