Module file_format

Expand description

Module containing helper methods for the various file formats See write.rs for write related helper methods

Modules§

arrow: ArrowFormat: Apache Arrow FileFormat abstractions
avro: AvroFormat Apache Avro FileFormat abstractions
csv: CsvFormat, Comma Separated Value (CSV) FileFormat abstractions
file_compression_type: File Compression type abstraction
json: JsonFormat: Line delimited JSON FileFormat abstractions
options: User facing options for the file formats readers
parquetparquet: ParquetFormat: Parquet FileFormat abstractions
write: Module containing helper methods/traits related to enabling write support for the various file formats

DefaultFileType: A container of FileFormatFactory which also implements FileType. This enables converting a dyn FileFormat to a dyn FileType. The former trait is a superset of the latter trait, which includes execution time relevant methods. FileType is only used in logical planning and only implements the subset of methods required during logical planning.

DeserializerOutput: Possible outputs of a BatchDeserializer.
FilePushdownSupport: An enum to distinguish between different states when determining if certain filters can be pushed down to file scanning

DEFAULT_SCHEMA_INFER_MAX_RECORD: Default max records to scan to infer the schema

BatchDeserializer: Trait defining a scheme for deserializing byte streams into structured data. Implementors of this trait are responsible for converting raw bytes into RecordBatch objects.
FileFormat: This trait abstracts all the file format specific implementations from the TableProvider. This helps code re-utilization across providers that support the same file formats.
FileFormatFactory: Factory for creating FileFormat instances based on session and command level options

file_type_to_format: Converts a FileType to a FileFormatFactory. Returns an error if the FileType cannot be downcasted to a DefaultFileType.
format_as_file_type: Converts a FileFormatFactory to a FileType