Module datasource

Expand description

DataFusion data sources: TableProvider and ListingTable

Modules§

cte_worktable: CteWorkTable implementation used for recursive queries
default_table_source: Default TableSource implementation used in DataFusion physical plans
dynamic_file: dynamic_file_schema contains an UrlTableFactory implementation that can create a ListingTable from the given url.
empty: EmptyTable useful for testing.
file_format: Module containing helper methods for the various file formats See write.rs for write related helper methods
listing: A table that uses the ObjectStore listing capability to get the list of files to process.
listing_table_factory: Factory for creating ListingTables with default options
memory
object_store: ObjectStoreRegistry holds all the object stores at Runtime with a scheme for each store. This allows the user to extend DataFusion with different storage systems such as S3 or HDFS and query data inside these systems.
physical_plan: Execution plans that read file formats
provider: Data source traits
schema_adapter: SchemaAdapter and SchemaAdapterFactory to adapt file-level record batches to a table schema.
sink: Execution plan for writing data to DataSinks
source: DataSource and DataSourceExec
stream: TableProvider for stream sources, such as FIFO files
view: View data source which uses a LogicalPlan as it’s input.

DefaultTableSource: Implements TableSource for a TableProvider
MemTable: In-memory data source for presenting a Vec<RecordBatch> as a data source that can be queried by DataFusion. This allows data to be pre-loaded into memory and then repeatedly queried without incurring additional file I/O overhead.
ViewTable: An implementation of TableProvider that uses another logical plan.

create_ordering: Converts logical sort expressions to physical sort expressions
provider_as_source: Wrap TableProvider in TableSource
source_as_provider: Attempt to downcast a TableSource to DefaultTableSource and access the TableProvider. This will only work with a TableSource created by DataFusion.