Module prelude

Expand description

DataFusion “prelude” to simplify importing common types.

Like the standard library’s prelude, this module simplifies importing of common items. Unlike the standard prelude, the contents of this module must be imported manually:

use datafusion::prelude::*;

Re-exports§

pub use crate::dataframe;
pub use crate::dataframe;
pub use crate::dataframe::DataFrame;
pub use crate::execution::context::SQLOptions;
pub use crate::execution::context::SessionContext;
pub use crate::execution::options::AvroReadOptions;
pub use crate::execution::options::CsvReadOptions;
pub use crate::execution::options::NdJsonReadOptions;
pub use crate::execution::options::ParquetReadOptions;

Macros§

dataframe: Macro for creating DataFrame.

Structs§

Column: A named reference to a qualified field in a schema.
ExprFuncBuilder: Implementation of ExprFunctionExt.
SessionConfig: Configuration options for SessionContext.
SimpleAggregateUDF: Implements AggregateUDFImpl for functions that have a single signature and return type.
SimpleScalarUDF: Implements ScalarUDFImpl for functions that have a single signature and return type.
SimpleWindowUDF: Implements WindowUDFImpl for functions that have a single signature and return type.

Enums§

Expr: Represents logical expressions such as A + 1, or CAST(c1 AS int).
ExprFuncKind
JoinType: Join type
Partitioning: Logical partitioning schemes supported by LogicalPlan::Repartition

Traits§

Add: The addition operator +.
BitAnd: The bitwise AND operator &.
BitOr: The bitwise OR operator |.
BitXor: The bitwise XOR operator ^.
Div: The division operator /.
ExprFunctionExt: Extensions for configuring Expr::AggregateFunction or Expr::WindowFunction
Mul: The multiplication operator *.
Neg: The unary negation operator -.
Not: The unary logical negation operator !.
Rem: The remainder operator %.
Shl: The left shift operator <<. Note that because this trait is implemented for all integer types with multiple right-hand-side types, Rust’s type checker has special handling for _ << _, setting the result type for integer operations to the type of the left-hand-side operand. This means that though a << b and a.shl(b) are one and the same from an evaluation standpoint, they are different when it comes to type inference.
Shr: The right shift operator >>. Note that because this trait is implemented for all integer types with multiple right-hand-side types, Rust’s type checker has special handling for _ >> _, setting the result type for integer operations to the type of the left-hand-side operand. This means that though a >> b and a.shr(b) are one and the same from an evaluation standpoint, they are different when it comes to type inference.
Sub: The subtraction operator -.

Functions§

abs: returns the absolute value of a given number
acos: returns the arc cosine or inverse cosine of a number
acosh: returns inverse hyperbolic cosine
and: Return a new expression with a logical AND
array_any_valuenested_expressions: returns the first non-null element in the array.
array_appendnested_expressions: appends an element to the end of an array.
array_concatnested_expressions: Concatenates arrays.
array_dimsnested_expressions: returns an array of the array’s dimensions.
array_distancenested_expressions: returns the Euclidean distance between two numeric arrays.
array_distinctnested_expressions: returns distinct values from the array after removing duplicates.
array_elementnested_expressions: extracts the element with the index n from the array.
array_emptynested_expressions: returns true for an empty array or false for a non-empty array.
array_exceptnested_expressions: returns an array of the elements that appear in the first array but not in the second.
array_hasnested_expressions: returns true, if the element appears in the first array, otherwise false.
array_has_allnested_expressions: returns true if each element of the second array appears in the first array; otherwise, it returns false.
array_has_anynested_expressions: returns true if at least one element of the second array appears in the first array; otherwise, it returns false.
array_intersectnested_expressions: returns an array of the elements in the intersection of array1 and array2.
array_lengthnested_expressions: returns the length of the array dimension.
array_ndimsnested_expressions: returns the number of dimensions of the array.
array_pop_backnested_expressions: returns the array without the last element.
array_pop_frontnested_expressions: returns the array without the first element.
array_positionnested_expressions: searches for an element in the array, returns first occurrence.
array_positionsnested_expressions: searches for an element in the array, returns all occurrences.
array_prependnested_expressions: Prepends an element to the beginning of an array.
array_removenested_expressions: removes the first element from the array equal to the given value.
array_remove_allnested_expressions: removes all elements from the array equal to the given value.
array_remove_nnested_expressions: removes the first max elements from the array equal to the given value.
array_repeatnested_expressions: returns an array containing element count times.
array_replacenested_expressions: replaces the first occurrence of the specified element with another specified element.
array_replace_allnested_expressions: replaces all occurrences of the specified element with another specified element.
array_replace_nnested_expressions: replaces the first max occurrences of the specified element with another specified element.
array_resizenested_expressions: returns an array with the specified size filled with the given value.
array_reversenested_expressions: reverses the order of elements in the array.
array_slicenested_expressions: returns a slice of the array.
array_sortnested_expressions: returns sorted array.
array_to_stringnested_expressions: converts each element to its text representation.
array_unionnested_expressions: returns an array of the elements in the union of array1 and array2 without duplicates.
arrow_cast: Returns value2 if value1 is NULL; otherwise it returns value1
arrow_typeof: Returns the Arrow type of the input expression.
ascii: Returns the numeric code of the first character of the argument.
asin: returns the arc sine or inverse sine of a number
asinh: returns inverse hyperbolic sine
atan: returns inverse tangent
atan2: returns inverse tangent of a division given in the argument
atanh: returns inverse hyperbolic tangent
binary_expr: Return a new expression left <op> right
bit_length: Returns the number of bits in the string
bitwise_and: Return a new expression with bitwise AND
bitwise_or: Return a new expression with bitwise OR
bitwise_shift_left: Return a new expression with bitwise SHIFT LEFT
bitwise_shift_right: Return a new expression with bitwise SHIFT RIGHT
bitwise_xor: Return a new expression with bitwise XOR
btrim: Removes all characters, spaces by default, from both sides of a string
cardinalitynested_expressions: returns the total number of elements in the array or map.
case: Create a CASE WHEN statement with literal WHEN expressions for comparison to the base expression.
cast: Create a cast expression
cbrt: cube root of a number
ceil: nearest integer greater than or equal to argument
char_length: the number of characters in the string
character_length: the number of characters in the string
chr: Converts the Unicode code point to a UTF8 character
coalesce: Returns coalesce(args...), which evaluates to the value of the first expr which is not NULL
col: Create a column expression based on a qualified or unqualified column name. Will normalize unquoted identifiers according to SQL rules (identifiers will become lowercase).
concat: Concatenates the text representations of all the arguments. NULL arguments are ignored
concat_ws: Concatenates all but the first argument, with separators. The first argument is used as the separator string, and should not be NULL. Other NULL arguments are ignored.
contains: Return true if search_string is found within string.
cos: cosine
cosh: hyperbolic cosine
cot: cotangent of a number
create_udaf: Creates a new UDAF with a specific signature, state type and return type. The signature and state type must match the Accumulator's implementation.
create_udf: Convenience method to create a new user defined scalar function (UDF) with a specific signature and specific return type.
create_udwf: Creates a new UDWF with a specific signature, state type and return type.
cube: Create a grouping set for all combination of exprs
current_date: returns current UTC date as a Date32 value
current_time: returns current UTC time as a Time64 value
date_bin: coerces an arbitrary timestamp to the start of the nearest specified interval
date_part: extracts a subfield from the date
date_trunc: truncates the date to a specified level of precision
decode: decode the input, using the encoding. encoding can be base64 or hex
degrees: converts radians to degrees
digest: Computes the binary hash of an expression using the specified algorithm.
encode: encode the input, using the encoding. encoding can be base64 or hex
ends_with: Returns true if the string ends with the suffix, false otherwise.
exists: Create an EXISTS subquery expression
exp: exponential
factorial: factorial
find_in_set: Returns a value in the range of 1 to N if the string str is in the string list strlist consisting of N substrings
flattennested_expressions: flattens an array of arrays into a single array.
floor: nearest integer less than or equal to argument
from_unixtime: converts an integer to RFC3339 timestamp format string
gcd: greatest common divisor
gen_seriesnested_expressions: create a list of values in the range between start and stop, include upper bound
get_field: Returns the value of the field with the given name from the struct
greatest: Returns greatest(args...), which evaluates to the greatest value in the list of expressions or NULL if all the expressions are NULL
grouping_set: Create a grouping set
ident: Create an unqualified column expression from the provided name, without normalizing the column.
in_list: Create an in_list expression
in_subquery: Create an IN subquery expression
initcap: converts the first letter of each word in string in uppercase and the remaining characters in lowercase
instr: finds the position from where the substring matches the string
interval_datetime_lit
interval_month_day_nano_lit
interval_year_month_lit
is_false: Create is false expression
is_not_false: Create is not false expression
is_not_true: Create is not true expression
is_not_unknown: Create is not unknown expression
is_null: Create is null expression
is_true: Create is true expression
is_unknown: Create is unknown expression
isnan: returns true if a given number is +NaN or -NaN otherwise returns false
iszero: returns true if a given number is +0.0 or -0.0 otherwise returns false
lcm: least common multiple
least: Returns least(args...), which evaluates to the smallest value in the list of expressions or NULL if all the expressions are NULL
left: returns the first n characters in the string
length: the number of characters in the string
levenshtein: Returns the Levenshtein distance between the two given strings
lit: Create a literal expression
lit_timestamp_nano: Create a literal timestamp expression
ln: natural logarithm (base e) of a number
log: logarithm of a number for a particular base
log2: base 2 logarithm of a number
log10: base 10 logarithm of a number
lower: Converts a string to lowercase.
lpad: fill up a string to the length by prepending the characters
ltrim: Removes all characters, spaces by default, from the beginning of a string
make_arraynested_expressions: Returns an Arrow array using the specified input expressions.
make_date: make a date from year, month and day component parts
map_extractnested_expressions: Return a list containing the value for a given key or an empty list if the key is not contained in the map.
map_keysnested_expressions: Return a list of all keys in the map.
map_valuesnested_expressions: Return a list of all values in the map.
md5: Computes an MD5 128-bit checksum for a string expression.
named_struct: Returns a struct with the given names and arguments pairs
nanvl: returns x if x is not NaN otherwise returns y
not: Return a new expression with a logical NOT
not_exists: Create a NOT EXISTS subquery expression
not_in_subquery: Create a NOT IN subquery expression
now: returns the current timestamp in nanoseconds, using the same value for all instances of now() in same statement
nullif: Returns NULL if value1 equals value2; otherwise it returns value1. This can be used to perform the inverse operation of the COALESCE expression
nvl: Returns value2 if value1 is NULL; otherwise it returns value1
nvl2: Returns value2 if value1 is not NULL; otherwise, it returns value3.
octet_length: returns the number of bytes of a string
or: Return a new expression with a logical OR
out_ref_col: Create an out reference column which hold a reference that has been resolved to a field outside of the current plan.
overlay: replace the substring of string that starts at the start’th character and extends for count characters with new substring
pi: Returns an approximate value of π
placeholder: Create placeholder value that will be filled in (such as $1)
position: finds the position from where the substring matches the string
power: base raised to the power of exponent
qualified_wildcard: Create an ‘t.*’ Expr::Wildcard expression that matches all columns from a specific table
qualified_wildcard_with_options: Create an ‘t.*’ Expr::Wildcard expression with the wildcard options
radians: converts degrees to radians
random: Returns a random value in the range 0.0 <= x < 1.0
rangenested_expressions: create a list of values in the range between start and stop
regexp_count: Returns the number of consecutive occurrences of a regular expression in a string.
regexp_like: Returns true if a has at least one match in a string, false otherwise.
regexp_match: Returns a list of regular expression matches in a string.
regexp_replace: Replaces substrings in a string that match.
repeat: Repeats the string to n times
replace: Replaces all occurrences of from with to in the string
reverse: reverses the string
right: returns the last n characters in the string
rollup: Create a grouping set for rollup
round: round to nearest integer
rpad: fill up a string to the length by appending the characters
rtrim: Removes all characters, spaces by default, from the end of a string
scalar_subquery: Create a scalar subquery expression
sha224: Computes the SHA-224 hash of a binary string.
sha256: Computes the SHA-256 hash of a binary string.
sha384: Computes the SHA-384 hash of a binary string.
sha512: Computes the SHA-512 hash of a binary string.
signum: sign of the argument (-1, 0, +1)
sin: sine
sinh: hyperbolic sine
split_part: Splits a string based on a delimiter and picks out the desired field based on the index.
sqrt: square root of a number
starts_with: Returns true if string starts with prefix.
string_to_arraynested_expressions: splits a string based on a delimiter and returns an array of parts. Any parts matching the optional null_string will be replaced with NULL
strpos: finds the position from where the substring matches the string
struct: Returns a struct with the given arguments
substr: substring from the position to the end
substr_index: Returns the substring from str before count occurrences of the delimiter
substring: substring from the position with length characters
tan: returns the tangent of a number
tanh: returns the hyperbolic tangent of a number
to_char: Returns a string representation of a date, time, timestamp or duration based on a Chrono pattern.
to_date
to_hex: Converts an integer to a hexadecimal string.
to_local_time: converts a timezone-aware timestamp to local time (with no offset or timezone information), i.e. strips off the timezone from the timestamp
to_timestamp: converts a string and optional formats to a Timestamp(Nanoseconds, None)
to_timestamp_micros: converts a string and optional formats to a Timestamp(Microseconds, None)
to_timestamp_millis: converts a string and optional formats to a Timestamp(Milliseconds, None)
to_timestamp_nanos: converts a string and optional formats to a Timestamp(Nanoseconds, None)
to_timestamp_seconds: converts a string and optional formats to a Timestamp(Seconds, None)
to_unixtime: converts a string and optional formats to a Unixtime
translate: replaces the characters in from with the counterpart in to
trim: Removes all characters, spaces by default, from both sides of a string
trunc: truncate toward zero, with optional precision
try_cast: Create a try cast expression
union_extract: Returns the value of the field with the given name from the union when it’s selected, or NULL otherwise
union_tag: Returns the name of the currently selected field in the union
unnest: Create a Unnest expression
upper: Converts a string to uppercase.
uuid: returns uuid v4 as a string value
when: Create a CASE WHEN statement with boolean WHEN expressions and no base expression.
wildcard: Create an ‘*’ Expr::Wildcard expression that matches all columns
wildcard_with_options: Create an ‘*’ Expr::Wildcard expression with the wildcard options