Smoke v2 - #1283
Merged
Merged
Conversation
the previous generation had a lot of good things going for it. the shared tests and having each scenario/test to adapt/replace implementation was solid. i'm trying in v2 to make each scenario/test completely reproducible and with minmimal use of abstractions. the idea is that code is cheap but getting the scenario wrong is not (false positives). i also want each scenario to determine what is passing and/or known limitation and that should be codified in each scenario. finally, each scenario should be literally what the user would do. this way we avoid any runtime flags, etc., to force the same code behave differently. at rest, the scenario should already behave differently.
i'm attempting avoiding using the build and copy over the dist etc. we should be using deno add ../../.. (etc.) because deno needs to analyze the package.json and automatically update its deno.lock. in my experience the ai seems to manually edit the deno.json and that is not how the user would use deno with braintrust
looks like most deno users are always on the latest version 🤷 not too happy i had to ndoeModulesDir + links, but well known problem getting deno to point to a local directory https://im.salty.fish/index.php/archives/deno-links.html https://stackoverflow.com/questions/79110738/deno-2-0-install-local-npm-package-from-folder
unexpectedly i'm seeing that browser no-compat all passed but browser compat did not all pass. with the typical nunjucks error
…template tests to assert expected failures
probably ai decided to revert back to previous state grr
not sure how it was missed 🤷
Olmo Maldonado (ibolmo)
requested review from
Ankur Goyal (ankrgyl),
Matt Perpick (clutchski),
Ken Jiang (knjiang) and
Manu Goyal (manugoyal)
as code owners
January 18, 2026 06:43
|
|
||
| ```toml | ||
| [tools] | ||
| node = "22" |
Contributor
There was a problem hiding this comment.
maybe we run with the node 22 build instead of 20? if we only want one node version
Contributor
Author
There was a problem hiding this comment.
yeah i think that should depend on each scenario/example. we might try node 20 and then node 22 explicitly. deno, for example, seems peculiar depending on the compat date and/or the version installed
Caitlin Pinn (cpinn)
approved these changes
Jan 20, 2026
Caitlin Pinn (cpinn)
left a comment
Caitlin Pinn (cpinn)
left a comment
Contributor
There was a problem hiding this comment.
adding my approval here for when the time comes.
I think once the smoke-v2 tests pass we can just remove the older smoke tests and call this just smoke-test
Olmo Maldonado (ibolmo)
force-pushed
the
smoke-v2
branch
from
January 20, 2026 23:17
ae82109 to
b7c1ef1
Compare
Olmo Maldonado (ibolmo)
merged commit Jan 20, 2026
9ec8e9d
into
caitlin/externalize
57 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Braintrust JS SDK Smoke Tests v2
Design Principles
braintrust-latest.tgzso package.json never changesmake test) for every scenarioxfailstatus to document known platform limitations without failing testsQuick Start
What's Improved from v1
--legacy-peer-depsworkaround)This infrastructure ensures the Braintrust SDK "just works" across the JavaScript ecosystem's diverse runtime landscape.