Skip to content

timely-v0.24.0

Latest
Compare
Choose a tag to compare
@github-actions github-actions released this 15 Sep 19:13
· 8 commits to master since this release
3d66d7f

This version of Timely has some exciting new features.

  • The Distributor trait offers a generalization of the Exchange type. It allows users to define custom distribution
    strategies for routing data across workers. The Exchange type requires a row-by-row distribution and uses
    container builders to form batches of data, which limits its flexibility. The Distributor trait, on the other hand,
    allows users to define their own behavior for distributing data, and does not need to work row-by-row.
    It only needs to uphold progress tracking requirements, which are that the amount of records on the input
    matches those on the output.
  • We split the Container trait into multiple traits. The Accountable trait indicates the number of records
    in a container for progress tracking and is the only required trait for a container.
    The iter and drain functions are part of the new IterContainer and DrainContainer traits.
    We do not require all containers to provide all of these traits, and it is mostly high-level operators that require
    a container to implement them. We alos removed the clear function from containers as it
    is not well-defined in all cases, for example for immutable data.
  • The Map trait offers a flat_map_builder function to accumulate flatmap logic.

Other

  • Introduce builder for flatmap operators (#704)
  • Container rework (#697)
  • Distributor trait (#700)