·
8 commits
to master
since this release
This version of Timely has some exciting new features.
- The
Distributor
trait offers a generalization of theExchange
type. It allows users to define custom distribution
strategies for routing data across workers. TheExchange
type requires a row-by-row distribution and uses
container builders to form batches of data, which limits its flexibility. TheDistributor
trait, on the other hand,
allows users to define their own behavior for distributing data, and does not need to work row-by-row.
It only needs to uphold progress tracking requirements, which are that the amount of records on the input
matches those on the output. - We split the
Container
trait into multiple traits. TheAccountable
trait indicates the number of records
in a container for progress tracking and is the only required trait for a container.
Theiter
anddrain
functions are part of the newIterContainer
andDrainContainer
traits.
We do not require all containers to provide all of these traits, and it is mostly high-level operators that require
a container to implement them. We alos removed theclear
function from containers as it
is not well-defined in all cases, for example for immutable data. - The
Map
trait offers aflat_map_builder
function to accumulate flatmap logic.