v2.0.0b34

Overview

b34 marks as one of the larges released in Flyte 2.0's SDK. It is production ready. Folks have scaled it to tens of thousands of tasks at rapid scale, able to download gigabytes of data and run large scale training on myraid architectures. This release takes us closed to the goal of Flyte 2.0's goals - one platform that is cohesive, simple, consistent, fast and end to end.
At a high level the most notable features are,

1. Serving - serve models, apps using the fast, consistent and simple API. This marks the beta release of the app devex. There will be some more minor changes, but many examples now show that it is possible to invoke services from tasks, tasks from services and services from services.
2. Support for custom context, that goes beyond regular inputs and can pass context data between tasks.
3. Tasks now have docs and description support.

Also welcome many new contributors. Follow the progress for Flyte 2.0 backend in the flyteorg/flyte - v2 branch. Remember as always, all of flyte-sdk can be run locally, so you do not need a backend!

What's Changed

• Call Apps from tasks. by @kumare3 in #349
• Kyle/sdk/feat/add version info by @k1sauce in #351
• Add support for --image to deploy by @kumare3 in #352
• Iceberg example using pyarrow tables by @kumare3 in #340
• Add custom context example by @jpvotta in #355
• fix(images): Support custom image URIs as strings in TaskEnvironment by @pingsutw in #348
• set deref_symlinks to True by @fiedlerNr9 in #359
• Add support tool-uv-index in the remote builder by @pingsutw in #357
• Minor updates by @wild-endeavor in #356
• Improve CLI doc gen by @ppiegaze in #362
• flyte serve and deployment updates by @kumare3 in #339
• update app replica range to (0, 1) by @cosmicBboy in #363
• [Refactor] Use ActionPhase from common phase proto by @machichima in #361
• Embedding recsys by @kumare3 in #360
• App deploy and serve refactored. with_servecontext by @kumare3 in #364
• Added H200 to available devices by @kumare3 in #367
• fix run and syncify typing by @cosmicBboy in #366
• Prep for new release b34 by @kumare3 in #368
• Add description to task deploy by @popojk in #314

New Contributors

• @k1sauce made their first contribution in #351

Full Changelog: v2.0.0b33...v2.0.0b34

v2.0.0b33

What's Changed

• Revert "Trigger time (#324)" by @kumare3 in #345
• pipe subdomain and custom domain to app idl by @cosmicBboy in #346
• skip final report flush when no report is available by @samhita-alla in #344
• Introduce _U_INSECURE env var by @pingsutw in #347

Full Changelog: v2.0.0b32...v2.0.0b33

v2.0.0b32: Alpha Apps, fixed regression on large fanout downloads

What's Changed

• Pass image builder project-domain scope and cache lookup scope by @EngHabu in #327
• Dataclass blob input example by @ursucarina in #336
• Alpha support: Flyte apps. Serve models, streamlit apps and much more! by @cosmicBboy in #212
• Standardize init behavior across most examples by @jpvotta in #338
• chore: Remove uv dev-dependencies from pyproject.toml by @pingsutw in #332
• supporting the file_name parameter in the new_remote function by @0yukali0 in #337
• ci: Increase PyPI publish wait time to 5 mins by @pingsutw in #333
• fix pythonpath deployment pattern example by @cosmicBboy in #342
• Comparison to s5cmd by @wild-endeavor in #292
• feat(maint_tools): Add bigquery extra to flyteplugins connector by @pingsutw in #334

New Contributors

• @0yukali0 made their first contribution in #337

Full Changelog: v2.0.0b31...v2.0.0b32

v2.0.0b31

What's Changed

• ci(publish.yml): Allow prerelease uploads to PyPI by @pingsutw in #331

Full Changelog: v2.0.0b30...v2.0.0b31

v2.0.0b30

What's Changed

• Don't use local file keyring when not in ipython by @wild-endeavor in #323
• Trigger time by @wild-endeavor in #324
• Scope image build runs to the same target project-domain by @EngHabu in #322
• Don't set default dst in with_source_folder by @pingsutw in #325
• Build flyte connector image by @pingsutw in #321
• Fix flyte serve cli by @pingsutw in #328
• Add example showing how to use time zones with cron schedule triggers by @jpvotta in #326
• Update trigger_serde by @pingsutw in #329
• spark transformer by @kevinliao852 in #289

Full Changelog: v2.0.0b29...v2.0.0b30

v2.0.0b29

What's Changed

• [ImageBuilder] Add project_install_mode arg to Poetry layer in Image spec by @popojk in #308
• Change setup to sync in dask plugin by @BarryWu0812 in #307
• remove trigger_time as default argument in triggers by @dansola in #315
• Allow relative root dir by @wild-endeavor in #316
• Type plugins by @pingsutw in #306
• Add task examples for various input types by @ursucarina in #318
• Set root handler level to debug by @wild-endeavor in #319
• Skip listing relative module files by @wild-endeavor in #317
• Add a lightweight JSON formatter by @wild-endeavor in #320

New Contributors

• @BarryWu0812 made their first contribution in #307
• @ursucarina made their first contribution in #318

Full Changelog: v2.0.0b28...v2.0.0b29

v2.0.0b28

What's Changed

• Use 3.13 to publish, not .x by @wild-endeavor in #312

Full Changelog: v2.0.0b27...v2.0.0b28

v2.0.0b27

What's Changed

• update ml gridsearch example by @cosmicBboy in #304
• introduce timezone for CRON by @fiedlerNr9 in #260
• Support CLI plugins by @wild-endeavor in #303
• Fix pyproject package deployment pattern readme by @davidmirror-ops in #305
• Bundle will follow symlinks by @kumare3 in #309
• Minor update to _file_is_in_directory by @wild-endeavor in #310

Full Changelog: v2.0.0b26...v2.0.0b27

v2.0.0b26

What's Changed

• Slack bot demo, that is crash-proof by @kumare3 in #280
• Slack bot faster by @kumare3 in #281
• Unit testing is natural by @kumare3 in #282
• Improve excepthook by @kumare3 in #284
• Updating listSecrets to have default page size of 10 by @pmahindrakar-oss in #285
• ImportError Diagnostics by @pingsutw in #288
• Support for installed function module extraction by @kumare3 in #290
• fix sync map by @samhita-alla in #293
• Fix error message formatting for clickable build link by @pingsutw in #291
• Incorrect Trigger type-hint and refactor module extraction by @kumare3 in #297
• Improve the error message when the image is not found in the image cache by @pingsutw in #298
• refactor(models): Use get_type_hints for param and return types by @pingsutw in #295
• Fast register uv workspace by @pingsutw in #283
• feat(dask): Add SchedulerPlugin for code bundle download by @pingsutw in #294
• Add missing allowed neuron quantities by @jeevb in #301
• Add addition attributes in RunSpec by @popojk in #217
• Update creation of image pull secrets by @wild-endeavor in #302
• Support for insecure endpoints by @mhotan in #299
• Connector Interface & Bigquery Connector by @pingsutw in #209

New Contributors

• @pmahindrakar-oss made their first contribution in #285
• @jeevb made their first contribution in #301
• @mhotan made their first contribution in #299

Full Changelog: v2.0.0b25...v2.0.0b26

[Breaking] v2.0.0b25

🛑 What breaks!

1. flyte.io.File/Dir apis have been updated. They have * for async and *_sync for sync apis.
2. flyte.io.File.open - returns memoryview/Bytes object which does not have decode. so you have to use bytes(obj).decode(...)
3. flyte.Image....with_uv_project() has a new flag, by default it will not install the code into the container. This means faster builds and reduced builds. But, if you want to install the package, set the flag accordingly

🤩 New features!

1. Triggers - cron and rate schedules
2. Torchrun (elastic) plugin
3. Support for tranium, gpus and tpus
4. Blazing fast data downloads. 5GB in 5s on a 10Gbps
5. Sync/async apis for file and dir
6. flyteidl2 usage
7. Faster flyte imports and cli
8. Lots of examples for HoF, UDFs
9. Context passing in tasks
10. Push/pull private images
11. Provides externally built images by name - Image.from_ref_name("my_image"). -- flyte run --image my_image="docker.io/...."

🤿 What's Changed

• Update dataframes example by @davidmirror-ops in #238
• Ignore dockerignore files when copying files into uv and poetry project by @popojk in #215
• ENG-158: Proto to flyteidl2 by @EngHabu in #178
• Improve error message for type engine by @pingsutw in #243
• higher order pattern examples by @dansola in #226
• Speed up imports by @kumare3 in #247
• Write higher order function that work with plain python udfs by @kumare3 in #249
• update resource tuner example by @cosmicBboy in #251
• [Feat] Enable setting image in cli by @machichima in #190
• catch FileNoteFoundError if git is not present by @cosmicBboy in #250
• chore: Remove unused connector init.py files by @pingsutw in #255
• Add require_project_and_domain decorator by @pingsutw in #256
• Correct syntax of run.wait() in examples by @ppiegaze in #258
• Pin idl2 by @wild-endeavor in #261
• Updates to _deploy logic and bump some actor versions by @wild-endeavor in #262
• Just make fmt by @wild-endeavor in #263
• Add PoetryProject and update IDL by @EngHabu in #174
• fix UvProjectHandler handle function by @popojk in #264
• Use idl2 proto in the dask plugin by @pingsutw in #257
• Pin flyteidl to a pre-release by @wild-endeavor in #267
• Use idl2 in the ray plugin by @pingsutw in #266
• [Breaking] Obstore parallel reader by @wild-endeavor in #253
• Parallel obstore should not use TaskGroup for 3.10 by @wild-endeavor in #269
• Add support for pushing and pulling private images in the remote builder by @pingsutw in #252
• make fmt by @wild-endeavor in #271
• Pyproject package Example by @kumare3 in #273
• Fix mypy tasks by @cosmicBboy in #265
• Trigger update by @kumare3 in #274
• Add debuggable support to tasks and plugins by @pingsutw in #240
• add context by @samhita-alla in #254
• UV project best practice by @pingsutw in #216
• Fixes logging of error path by @kumare3 in #276
• create all types example by @squiishyy in #160
• refactor(Image): Default project_install_mode to dependencies_only by @pingsutw in #277

New Contributors

• @davidmirror-ops made their first contribution in #238
• @EngHabu made their first contribution in #178
• @dansola made their first contribution in #226

Full Changelog: v2.0.0b24...v2.0.0b25