Skip to content

Commit 1fd07b0

Browse files
kev1nclaude
andcommitted
test(smoke): published-artifact smoke against prod (npm + PyPI)
Installs @getanyapi/sdk from npm and getanyapi from PyPI into throwaway dirs (never the local source), mints a capped ephemeral key via the public /agent/signup, makes one real reddit.search call, and asserts the packaged export surface + a well-formed, charged envelope. Catches packaging bugs (missing files, broken exports) the mock-only suite cannot. - smoke/npm-smoke.mjs, smoke/pypi-smoke.py: the per-SDK smokes - scripts/smoke.sh: local runner (VERSION= or positional; installs from registry) - .github/workflows/smoke.yml: workflow_dispatch + nightly only, no secrets, not a required check (registry flakiness can never block a PR) Validated live against v0.1.0: both SDKs PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 7a9818f commit 1fd07b0

5 files changed

Lines changed: 515 additions & 0 deletions

File tree

.github/workflows/smoke.yml

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
name: smoke
2+
3+
# Published-artifact smoke: install the SDKs FROM THE REGISTRIES and make one real
4+
# production call each, to catch packaging bugs (missing files, broken exports)
5+
# the mock-only CI suite cannot.
6+
#
7+
# Dispatch + nightly ONLY. This never runs on push/pull_request and is NOT a
8+
# required check, so upstream/registry flakiness can never block a PR. No secrets:
9+
# each smoke mints its own capped key via the public /agent/signup endpoint.
10+
11+
on:
12+
workflow_dispatch:
13+
inputs:
14+
version:
15+
description: "Published version to smoke (e.g. 0.1.0), or 'latest'."
16+
required: false
17+
default: "latest"
18+
type: string
19+
schedule:
20+
# Nightly at 08:00 UTC.
21+
- cron: "0 8 * * *"
22+
23+
jobs:
24+
npm-smoke:
25+
name: npm @getanyapi/sdk (published-artifact)
26+
runs-on: ubuntu-latest
27+
steps:
28+
- uses: actions/checkout@v4
29+
- uses: actions/setup-node@v4
30+
with:
31+
node-version: 20
32+
- name: install published package into a clean temp dir + run smoke
33+
env:
34+
VERSION: ${{ github.event.inputs.version || 'latest' }}
35+
run: |
36+
set -euo pipefail
37+
spec="@getanyapi/sdk"
38+
[ "$VERSION" != "latest" ] && spec="@getanyapi/sdk@$VERSION"
39+
tmp="$(mktemp -d)"
40+
cd "$tmp"
41+
npm init -y >/dev/null
42+
npm install "$spec"
43+
cp "$GITHUB_WORKSPACE/smoke/npm-smoke.mjs" ./npm-smoke.mjs
44+
node npm-smoke.mjs
45+
46+
pypi-smoke:
47+
name: pypi getanyapi (published-artifact)
48+
runs-on: ubuntu-latest
49+
steps:
50+
- uses: actions/checkout@v4
51+
- uses: actions/setup-python@v5
52+
with:
53+
python-version: "3.11"
54+
- name: install published package into a fresh venv + run smoke
55+
env:
56+
VERSION: ${{ github.event.inputs.version || 'latest' }}
57+
run: |
58+
set -euo pipefail
59+
spec="getanyapi"
60+
[ "$VERSION" != "latest" ] && spec="getanyapi==$VERSION"
61+
tmp="$(mktemp -d)"
62+
python -m venv "$tmp/venv"
63+
"$tmp/venv/bin/python" -m pip install --upgrade pip
64+
"$tmp/venv/bin/python" -m pip install "$spec"
65+
cp "$GITHUB_WORKSPACE/smoke/pypi-smoke.py" "$tmp/pypi-smoke.py"
66+
cd "$tmp"
67+
"$tmp/venv/bin/python" pypi-smoke.py

README.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,25 @@ pip install -e "packages/python[dev]"
108108
cd packages/python && pyright && mypy && pytest
109109
```
110110

111+
### Published-artifact smoke
112+
113+
`pnpm check` and the CI suite are mock-only: they never touch the registries. The
114+
published-artifact smoke fills that gap. It installs BOTH SDKs FROM the registries (npm +
115+
PyPI) into throwaway temp dirs outside the repo, mints an ephemeral capped key via the public
116+
`/agent/signup` endpoint (no secrets needed), and makes ONE real production call through each
117+
(a `$0.001` `reddit.search`), then asserts a well-formed, non-empty envelope. This catches
118+
packaging bugs, missing files, broken exports or types, that source-tree tests cannot see.
119+
120+
```bash
121+
bash scripts/smoke.sh # smoke the latest published version
122+
VERSION=0.1.0 bash scripts/smoke.sh
123+
```
124+
125+
It exits nonzero if either SDK fails and prints a per-SDK PASS/FAIL summary. In CI it is a
126+
standalone workflow (`.github/workflows/smoke.yml`) that runs nightly (08:00 UTC) and on manual
127+
dispatch (with an optional `version` input). It is intentionally NOT wired into `ci.yml`,
128+
`release.yml`, or branch protection, so registry or upstream flakiness never blocks a PR.
129+
111130
## Releasing
112131

113132
Releases are automated from the live catalog. Two workflows drive it:

scripts/smoke.sh

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
#!/usr/bin/env bash
2+
# Published-artifact smoke runner.
3+
#
4+
# Installs BOTH SDKs FROM THE REGISTRIES (npm + PyPI) into throwaway temp dirs
5+
# OUTSIDE this repo, then runs one real production call through each. This catches
6+
# packaging bugs (missing files, broken exports/types) that the repo's mock-only
7+
# test suite cannot: it never resolves the local workspace source.
8+
#
9+
# Usage:
10+
# bash scripts/smoke.sh # test the "latest" published version
11+
# VERSION=0.1.0 bash scripts/smoke.sh
12+
# bash scripts/smoke.sh 0.1.0 # positional version also works
13+
#
14+
# Exits nonzero if EITHER SDK smoke fails. Prints a per-SDK PASS/FAIL summary.
15+
set -uo pipefail
16+
17+
VERSION="${1:-${VERSION:-latest}}"
18+
19+
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
20+
SMOKE_DIR="$REPO_ROOT/smoke"
21+
22+
echo "== Published-artifact smoke (version: $VERSION) =="
23+
24+
npm_result="SKIP"
25+
pypi_result="SKIP"
26+
27+
# --- npm --------------------------------------------------------------------
28+
run_npm() {
29+
local spec="@getanyapi/sdk"
30+
if [ "$VERSION" != "latest" ]; then
31+
spec="@getanyapi/sdk@$VERSION"
32+
fi
33+
local tmp
34+
tmp="$(mktemp -d "${TMPDIR:-/tmp}/anyapi-npm-smoke.XXXXXX")"
35+
echo "-- npm: installing $spec into $tmp"
36+
(
37+
cd "$tmp" || exit 1
38+
npm init -y >/dev/null 2>&1 || exit 1
39+
# Install strictly from the registry; never link the local workspace.
40+
npm install "$spec" >/dev/null 2>&1 || exit 1
41+
) || { echo "npm install failed"; rm -rf "$tmp"; return 1; }
42+
cp "$SMOKE_DIR/npm-smoke.mjs" "$tmp/npm-smoke.mjs"
43+
( cd "$tmp" && node npm-smoke.mjs )
44+
local rc=$?
45+
rm -rf "$tmp"
46+
return $rc
47+
}
48+
49+
# --- pypi -------------------------------------------------------------------
50+
run_pypi() {
51+
local spec="getanyapi"
52+
if [ "$VERSION" != "latest" ]; then
53+
spec="getanyapi==$VERSION"
54+
fi
55+
local tmp
56+
tmp="$(mktemp -d "${TMPDIR:-/tmp}/anyapi-pypi-smoke.XXXXXX")"
57+
echo "-- pypi: installing $spec into a fresh venv at $tmp"
58+
local py="python3"
59+
command -v python3 >/dev/null 2>&1 || py="python"
60+
"$py" -m venv "$tmp/venv" || { echo "venv create failed"; rm -rf "$tmp"; return 1; }
61+
# Install strictly from the registry into the isolated venv.
62+
"$tmp/venv/bin/python" -m pip install --quiet --upgrade pip >/dev/null 2>&1
63+
if ! "$tmp/venv/bin/python" -m pip install --quiet "$spec" >/dev/null 2>&1; then
64+
echo "pip install failed"; rm -rf "$tmp"; return 1
65+
fi
66+
cp "$SMOKE_DIR/pypi-smoke.py" "$tmp/pypi-smoke.py"
67+
( cd "$tmp" && "$tmp/venv/bin/python" pypi-smoke.py )
68+
local rc=$?
69+
rm -rf "$tmp"
70+
return $rc
71+
}
72+
73+
if run_npm; then npm_result="PASS"; else npm_result="FAIL"; fi
74+
echo
75+
if run_pypi; then pypi_result="PASS"; else pypi_result="FAIL"; fi
76+
77+
echo
78+
echo "== Summary (version: $VERSION) =="
79+
echo " npm @getanyapi/sdk : $npm_result"
80+
echo " pypi getanyapi : $pypi_result"
81+
82+
if [ "$npm_result" = "PASS" ] && [ "$pypi_result" = "PASS" ]; then
83+
exit 0
84+
fi
85+
exit 1

smoke/npm-smoke.mjs

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
// Published-artifact smoke for @getanyapi/sdk (npm).
2+
//
3+
// Installs are done by scripts/smoke.sh / the workflow: this file resolves the
4+
// package from the CWD's node_modules (the INSTALLED artifact), NEVER a relative
5+
// path into the repo source. It exercises the real packaging: exports, types
6+
// (via the published d.ts at build time is separate; here we assert the runtime
7+
// surface), and one real production call.
8+
//
9+
// Flow: mint an ephemeral key via agentSignup() -> construct AnyAPI -> one live
10+
// reddit.search call -> assert a well-formed, non-empty envelope. Exit nonzero on
11+
// any failure with a readable report.
12+
13+
import { createRequire } from "node:module";
14+
15+
const require = createRequire(`${process.cwd()}/`);
16+
17+
/** Resolve @getanyapi/sdk from the CWD, so we test the installed artifact. */
18+
async function loadSdk() {
19+
// Resolve the package's own entry from the CWD's node_modules.
20+
const entry = require.resolve("@getanyapi/sdk");
21+
const mod = await import(entry);
22+
return mod;
23+
}
24+
25+
/** Retry only genuine transport flakiness, never assertion failures. */
26+
async function withRetry(label, fn, attempts = 3) {
27+
let lastErr;
28+
for (let i = 1; i <= attempts; i++) {
29+
try {
30+
return await fn();
31+
} catch (err) {
32+
// A FlakeError is our signal to retry; anything else fails immediately.
33+
if (!(err instanceof FlakeError) || i === attempts) {
34+
throw err;
35+
}
36+
lastErr = err;
37+
const backoffMs = 500 * i;
38+
console.error(
39+
`[retry] ${label} attempt ${i} failed (${err.message}); retrying in ${backoffMs}ms`,
40+
);
41+
await new Promise((r) => setTimeout(r, backoffMs));
42+
}
43+
}
44+
throw lastErr;
45+
}
46+
47+
/** Marks a transient transport error worth retrying. */
48+
class FlakeError extends Error {}
49+
50+
/** True when an error looks like transient network trouble. */
51+
function isTransient(err, sdk) {
52+
if (err instanceof sdk.ConnectionError || err instanceof sdk.TimeoutError) {
53+
return true;
54+
}
55+
if (err instanceof sdk.RateLimitedError) {
56+
return true;
57+
}
58+
const msg = String(err && err.message).toLowerCase();
59+
return (
60+
msg.includes("fetch failed") ||
61+
msg.includes("network") ||
62+
msg.includes("econnreset") ||
63+
msg.includes("timed out")
64+
);
65+
}
66+
67+
function assert(cond, message) {
68+
if (!cond) {
69+
throw new Error(`assertion failed: ${message}`);
70+
}
71+
}
72+
73+
/** Extract the posts array from either a bare or found-data run envelope. */
74+
function postsOf(res) {
75+
const output = res && res.output;
76+
if (output && typeof output === "object" && "found" in output) {
77+
return output.found && output.data ? output.data.posts : undefined;
78+
}
79+
return output ? output.posts : undefined;
80+
}
81+
82+
async function main() {
83+
const sdk = await loadSdk();
84+
85+
// Assert the packaged surface exports the symbols consumers rely on. A missing
86+
// export here is exactly the packaging bug this smoke exists to catch.
87+
for (const name of ["AnyAPI", "agentSignup", "unwrap", "AnyAPIError"]) {
88+
assert(typeof sdk[name] !== "undefined", `export "${name}" is missing from @getanyapi/sdk`);
89+
}
90+
assert(typeof sdk.AnyAPI === "function", "AnyAPI is not a constructor");
91+
assert(typeof sdk.agentSignup === "function", "agentSignup is not a function");
92+
93+
// 1. Mint an ephemeral, capped key. No stored secret.
94+
const signup = await withRetry("agentSignup", async () => {
95+
try {
96+
return await sdk.agentSignup({ label: "sdk-published-smoke" });
97+
} catch (err) {
98+
if (isTransient(err, sdk)) throw new FlakeError(err.message);
99+
throw err;
100+
}
101+
});
102+
assert(typeof signup.secret === "string" && signup.secret.length > 0, "signup.secret is empty");
103+
assert(typeof signup.capUsd === "number", "signup.capUsd is not a number");
104+
console.log(`[ok] minted key (capUsd=$${signup.capUsd})`);
105+
106+
// 2. Construct a client with the minted secret.
107+
const client = new sdk.AnyAPI({ apiKey: signup.secret });
108+
109+
// 3. One real, cheap, live data call. reddit.search is $0.001 and non-mock.
110+
// If the freshly minted cap blocks it (402), fall back to the cheapest
111+
// known live SKU that returns a real 200.
112+
let sku = "reddit.search";
113+
let res;
114+
try {
115+
res = await withRetry("reddit.search", async () => {
116+
try {
117+
return await client.reddit.search({ query: "mechanical keyboard", sort: "top" });
118+
} catch (err) {
119+
if (isTransient(err, sdk)) throw new FlakeError(err.message);
120+
throw err;
121+
}
122+
});
123+
} catch (err) {
124+
if (err instanceof sdk.InsufficientBalanceError) {
125+
// Cap/credit too small for reddit.search: fall back to the cheapest live SKU.
126+
sku = "reddit.subreddit_details";
127+
console.error(`[fallback] reddit.search blocked by cap (402); trying ${sku}`);
128+
res = await withRetry(sku, async () => {
129+
try {
130+
return await client.reddit.subredditDetails({ subreddit: "programming" });
131+
} catch (e) {
132+
if (isTransient(e, sdk)) throw new FlakeError(e.message);
133+
throw e;
134+
}
135+
});
136+
} else {
137+
throw err;
138+
}
139+
}
140+
141+
// 4. Assert the envelope is well-formed and the call was a real success.
142+
assert(res && typeof res === "object", "no response object");
143+
assert(res.provider === "AnyAPI", `provider must be "AnyAPI", got ${JSON.stringify(res.provider)}`);
144+
assert(typeof res.costUsd === "number" && res.costUsd >= 0, "costUsd missing/invalid (not a real charged call)");
145+
146+
if (sku === "reddit.search") {
147+
const posts = postsOf(res);
148+
assert(Array.isArray(posts), "output.posts is not an array");
149+
assert(posts.length > 0, "output.posts is empty (no real data returned)");
150+
const first = posts[0];
151+
assert(typeof first.title === "string" && first.title.length > 0, "first post has no title");
152+
assert(typeof first.subreddit === "string", "first post has no subreddit");
153+
console.log(
154+
`[ok] ${sku} -> ${posts.length} posts, costUsd=$${res.costUsd}, first="${first.title.slice(0, 60)}"`,
155+
);
156+
} else {
157+
// Fallback SKU: found-data envelope with a typed subreddit record.
158+
const data = sdk.unwrap(res);
159+
assert(data && typeof data.name === "string" && data.name.length > 0, "subreddit details missing name");
160+
console.log(`[ok] ${sku} -> r/${data.name}, costUsd=$${res.costUsd}`);
161+
}
162+
163+
console.log(`PASS npm @getanyapi/sdk: live ${sku} call succeeded against production.`);
164+
}
165+
166+
main().catch((err) => {
167+
console.error(`FAIL npm @getanyapi/sdk: ${err && err.message ? err.message : err}`);
168+
if (err && err.stack) console.error(err.stack);
169+
process.exit(1);
170+
});

0 commit comments

Comments
 (0)