Module datafusion::datasource

DataFusion data sources

Re-exports

pub use self::datasource::TableProvider;

pub use self::datasource::TableType;

pub use self::memory::MemTable;

Modules

datasource

Data source traits

empty

An empty plan that is usefull for testing and generating plans without mapping them to actual data.

file_format

Module containing helper methods for the various file formats

listing

A table that uses the ObjectStore listing capability to get the list of files to process.

memory

In-memory data source for presenting a Vec as a data source that can be queried by DataFusion. This allows data to be pre-loaded into memory and then repeatedly queried without incurring additional file I/O overhead.

object_store

Object Store abstracts access to an underlying file/object storage.

Structs

PartitionedFile

A single file that should be read, along with its schema, statistics and partition column values that need to be appended to each row.

Functions

get_statistics_with_limit

Get all files as well as the file level summary statistics (no statistic for partition columns). If the optional limit is provided, includes only sufficient files. Needed to read up to limit number of rows. TODO fix case where num_rows and total_byte_size are not defined (stat should be None instead of Some(0))

Type Definitions

PartitionedFileStream

Stream of files get listed from object store