dev nightly tests #801

Workflow file for this run

.github/workflows/dev-nightly-tests.yml at 918a7b7

	# Periodic tests of latest dev branch
	#
	# This is for early detection of breaking changes.
	#

	name: dev nightly tests

	# Controls when the workflow will run
	on:
	schedule:
	- cron: "0 5 * * *" # every day 5AM

	workflow_dispatch:
	inputs:
	force_build:
	description: "Force rebuild + tests even when dist digests match"
	type: boolean
	default: false

	# Least-privilege default for every job; 'test' overrides to write because it
	# commits the refreshed dist package back to dev.
	permissions:
	contents: read

	jobs:
	test:
	name: Test ${{ matrix.os }}
	runs-on: ${{ matrix.os }}
	timeout-minutes: 90
	permissions:
	contents: write
	env:
	# Scheduled runs leave this '0'; a manual dispatch can force package.py
	# to rebuild and re-test everything despite matching dist digests.
	PACKAGE_FORCE_BUILD: ${{ inputs.force_build && '1' \|\| '0' }}
	strategy:
	matrix:
	os: [ubuntu-latest, windows-2022]

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	- name: Install Python dependencies
	run: pip install PyGithub

	- name: Prepare shell envs (Linux)
	if: runner.os == 'Linux'
	shell: bash
	run: \|
	echo "PYTHON=python3" >> $GITHUB_ENV

	# Restore the packages referenced by dist/digests/ BEFORE package.py runs.
	# The packages are no longer committed, so a fresh checkout has none of them,
	# and package.py needs them on disk for two separate reasons:
	# 1. is_build_skipping_allowed() hashes the local file. With no file there is
	# no skip, so main would REBUILD instead of reusing the artifact dev built
	# and tested -- silently breaking the property this whole design protects.
	# 2. the compare_*_files helpers diff against the previous package to decide
	# whether anything actually changed. With no baseline every asset looks new
	# and churns on every run.
	# --allow-missing: a pool miss here is self-healing (package.py rebuilds and the
	# push step re-uploads), so it must not turn a recoverable state into a red build.
	- name: Fetch dist packages from the asset pool (Linux)
	if: runner.os == 'Linux'
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	shell: bash
	run: $PYTHON $GITHUB_WORKSPACE/scripts/dist-pool.py pull --allow-missing

	- name: Install dist build tooling (Linux)
	if: runner.os == 'Linux'
	uses: ./.github/actions/apt-install
	with:
	# One install per line, matching the grouping this job has always
	# used; merging them breaks gcc-13-*-linux-gnu resolution.
	packages: \|
	automake libtool autogen
	openjdk-21-jdk
	gcc-multilib
	gcc-i686-linux-gnu
	gcc-aarch64-linux-gnu

	- name: Build dist assets (Linux)
	if: runner.os == 'Linux'
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	shell: bash
	run: \|
	$PYTHON $GITHUB_WORKSPACE/scripts/package.py
	$PYTHON $GITHUB_WORKSPACE/scripts/test-dist.py

	- name: Prepare shell envs (Windows x64)
	if: runner.os == 'Windows'
	shell: cmd
	run: \|
	:: call "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe" -latest -products * -requires Microsoft.Component.MSBuild -find MSBuild\**\Bin\MSBuild.exe > msbuild_path.txt
	:: set /p MSBUILD_PATH=<msbuild_path.txt
	:: set VCVARSALL=%MSBUILD_PATH%\..\..\..\..\VC\Auxiliary\Build\vcvarsall.bat
	:: echo VCVARSALL=%VCVARSALL% >> %GITHUB_ENV%
	echo PYTHON=python >> %GITHUB_ENV%

	# NOTE: one command per step (or && chains). A multi-line `shell: cmd`
	# script propagates only the LAST command's exit code, which masked
	# every failure in this job for months (wix EULA errors, full MSVC
	# build failures, dead MSI packaging) while the job stayed green.
	# WiX is version-pinned for determinism (an unpinned install is what
	# silently jumped to v7 in April and broke packaging).
	- name: Install WiX (Windows x64)
	if: runner.os == 'Windows'
	shell: cmd
	run: dotnet tool install --global wix --version 7.0.0 && echo %USERPROFILE%\.dotnet\tools>>%GITHUB_PATH%

	# TA-Lib generates no revenue, so the Open Source Maintenance Fee does
	# not apply (the OSMF exempts organizations under USD 10k/year), but
	# WiX v7 still requires explicit EULA acceptance (WIX7015 otherwise).
	# Accepted per project-maintainer decision, 2026-07-03.
	- name: Accept WiX OSMF EULA (Windows x64)
	if: runner.os == 'Windows'
	shell: cmd
	run: wix eula accept wix7

	- name: Add WiX UI extension (Windows x64)
	if: runner.os == 'Windows'
	shell: cmd
	run: wix extension add --global WixToolset.UI.wixext/7.0.0 && wix extension list --global

	# See the Linux fetch step above for why this must precede package.py.
	- name: Fetch dist packages from the asset pool (Windows x64)
	if: runner.os == 'Windows'
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	shell: cmd
	run: '%PYTHON% %GITHUB_WORKSPACE%\scripts\dist-pool.py pull --allow-missing'

	- name: Build dist assets (Windows x64)
	if: runner.os == 'Windows'
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	shell: cmd
	run: '%PYTHON% %GITHUB_WORKSPACE%\scripts\package.py'

	- name: Test dist assets (Windows x64)
	if: runner.os == 'Windows'
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	shell: cmd
	run: '%PYTHON% %GITHUB_WORKSPACE%\scripts\test-dist.py'

	# Upload the freshly built packages into the content-addressed asset pool.
	#
	# ORDER IS LOAD-BEARING: this runs BEFORE the digests that reference the
	# packages are committed. The reverse order can commit a digest naming a
	# package that was never uploaded, and that state is not recoverable by
	# retry -- the only way out is a full rebuild + re-test. For the same
	# reason this step must never be continue-on-error: a failed upload has to
	# stop the run before anything is committed.
	#
	# Idempotent: assets are named by content hash, so re-running uploads
	# nothing new, and a race with the other matrix leg resolves to success.
	- name: Push dist packages to the asset pool (Linux)
	if: runner.os == 'Linux'
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	shell: bash
	run: $PYTHON $GITHUB_WORKSPACE/scripts/dist-pool.py push

	- name: Push dist packages to the asset pool (Windows x64)
	if: runner.os == 'Windows'
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	shell: cmd
	run: '%PYTHON% %GITHUB_WORKSPACE%\scripts\dist-pool.py push'

	- name: Check for changes and commit if any
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	shell: bash
	run: \|
	# Keep the website install page (website/src/install/c/README.md) pointed
	# at the latest published GitHub release, committing it alongside the dist
	# assets so the repo and the deployed website always match. Linux-only so
	# it is committed once; idempotent and a no-op if the GitHub API is
	# unavailable (the update is eventual -- it retries next run).
	if [ "${{ matrix.os }}" = "ubuntu-latest" ]; then
	$PYTHON $GITHUB_WORKSPACE/scripts/sync-website.py \|\| true
	fi
	dist_changes=$(git status --porcelain dist/)
	ta_common_changes=$(git status --porcelain include/ta_common.h)
	website_changes=$(git status --porcelain website/src/install/c/README.md)
	if [ -n "$dist_changes" ] \|\| [ -n "$ta_common_changes" ] \|\| [ -n "$website_changes" ]; then
	git config --global user.name "github-actions[bot]"
	git config --global user.email "github-actions[bot]@users.noreply.github.com"
	echo "Changes detected by git status:"
	if [ -n "$dist_changes" ]; then
	echo "$dist_changes"
	git add dist/
	fi
	if [ -n "$ta_common_changes" ]; then
	echo "$ta_common_changes"
	git add include/ta_common.h
	fi
	if [ -n "$website_changes" ]; then
	echo "$website_changes"
	git add website/src/install/c/README.md
	fi

	git commit -m "Update dist package (from ${{ matrix.os }})"

	# Check for unstaged changes and log them.
	# (this help debugging because it should not happen)
	if [ -n "$(git status --porcelain)" ]; then
	echo "Unstaged changes detected:"
	git status --porcelain
	fi

	# Retry mechanism for handling concurrent pushes
	for i in {1..5}; do
	git pull --rebase --strategy-option=ours origin dev && git push origin dev && break \|\| sleep 10
	done
	else
	echo "No changes detected in dist directory"
	fi

	# First-ever clang + ARM64 + macOS coverage of the generated C: build with
	# the default Apple toolchain and run the full C reference suite. Kept
	# separate from the dist 'test' job (macOS produces no dist assets); the
	# C hero philosophy applies — this needs no Rust/Java/.NET toolchain.
	test-macos:
	name: Test macos-latest (clang, ARM64)
	runs-on: macos-latest
	timeout-minutes: 90

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	- name: Build (CMake, AppleClang)
	shell: bash
	run: \|
	mkdir -p cmake-build && cd cmake-build
	cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_DEV_TOOLS=ON
	cmake --build . -j 3

	- name: Run C regression suite
	shell: bash
	run: cd cmake-build/bin && ./ta_regtest

	# Cross-language codegen verification — runs AFTER the C tests (needs: test).
	# C is the hero: the 'test' job above validates the C library with a C-only
	# toolchain (CMake/autotools never invoke cargo, so ta_regtest needs no Rust
	# toolchain). Only this job adds the Rust/Java/.NET toolchains, builds the
	# JSON-RPC language servers plus the frozen reference server (ta_ref_serve,
	# from the pinned reference tag), and runs ta_regtest --codegen:
	# 161 functions x 4 languages diffed against the reference, including the
	# non-default-parameter ref differential sweep. POSIX-only (the server
	# harness uses fork/exec), hence no Windows leg.
	#
	# --codegen runs the C reference tests in the same process as the codegen
	# sweep, which is what lets server_verify verify the HAND-WRITTEN tests
	# against all four servers bitwise (it rides them). Re-running the suite the
	# hero 'test' job already ran costs ~6s; that is what buys the hand-written
	# tier a cross-language check.
	cross-language:
	name: "xlang: all langs vs pre-cutover C"
	needs: test
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev
	fetch-depth: 0 # ta_ref_serve builds from the pinned reference tag

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	- name: Install JDK
	uses: ./.github/actions/apt-install
	with:
	packages: openjdk-21-jdk

	- name: Install .NET SDK
	uses: actions/setup-dotnet@v6
	with:
	dotnet-version: "10.0.x"

	- name: Build servers and run cross-language verification
	shell: bash
	run: \|
	# Rust toolchain and cmake are preinstalled on ubuntu-latest.
	# Benchmarks are skipped: CI timing is noise.
	python3 scripts/regtest.py --no-perftest --no-direct-bench --codegen

	# The benchmark CORPUS, not the benchmark. --verify-corpus checks every
	# shape is reproducible and produces valid OHLC — deterministic, so unlike
	# the timings it belongs in CI. It rides this job because regtest.py's
	# build step already put ta_bench in bin/. A shape that stops generating
	# valid data silently changes what every bench measures (#147).
	- name: Verify the benchmark input corpus
	shell: bash
	run: cd bin && ./ta_bench --verify-corpus --points=100000

	# Synthetic-function gate. The cross-language job above verifies the SHIPPED
	# functions; this one verifies generator surface no shipped function uses
	# (bitwise operators, integer truthiness, do-while, switch-on-expression, ...).
	# scripts/synth_gate.py copies the committed template functions from
	# ta_codegen/generator/input_synth/ into ta_codegen/input/ inside a throwaway git
	# worktree, regenerates ALL backends there, and runs the usual gates scoped
	# to the SYNTH family: --codegen (stream_verify / OpenAndFill,
	# batch-vs-stream bitwise in all four servers) and --xlang-hash (batch
	# parity, Rust/Java/C# vs the in-process C golden, bitwise). The script
	# asserts the SYNTH functions were actually swept in every leg, so a
	# refactor that silently stops enumerating them fails instead of passing an
	# empty run. Nothing synthetic ever lands in the shipped tree.
	synth-gate:
	name: "synth-gate: unshipped generator surface, all langs"
	needs: test
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev
	fetch-depth: 0 # ta_ref_serve builds from the pinned reference tag

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	- name: Install JDK
	uses: ./.github/actions/apt-install
	with:
	packages: openjdk-21-jdk

	- name: Install .NET SDK
	uses: actions/setup-dotnet@v6
	with:
	dotnet-version: "10.0.x"

	- name: Run the synthetic cross-language gate
	shell: bash
	run: \|
	# Rust toolchain and cmake are preinstalled on ubuntu-latest.
	python3 scripts/synth_gate.py

	# Debug-profile Rust codegen gate. The release cross-language job above wraps a
	# usize overflow silently (the wrapped value is dead), so it cannot see the
	# IMI/APO/PPO-class arithmetic-overflow bugs: those only manifest as a debug
	# panic. This job rebuilds the Rust server with the debug profile (overflow
	# checks on) and runs the codegen edge-range sweep (test_codegen.c
	# run_edge_range_sweep) against it, so any such overflow becomes a hard
	# failure. Rust-only (no JDK/.NET), gated on the C hero job.
	#
	# The BATCH tier is what the sweep covers; the second step extends the same
	# debug binary to the STREAM tier (Open/Update/Peek/Close, rings, sub-handles),
	# which the sweep never calls. Same cargo invocation, so the build is already
	# warm — it costs one extra pipe, not a second job.
	cross-language-rust-debug:
	name: Rust tests (debug profile — overflow gate)
	needs: test
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev
	fetch-depth: 0 # ta_ref_serve builds from the pinned reference tag

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	- name: Install PyYAML
	shell: bash
	run: pip install pyyaml

	- name: Build debug Rust server and run codegen overflow gate
	shell: bash
	run: \|
	# Rust toolchain and cmake are preinstalled on ubuntu-latest; no JDK/.NET
	# (Rust-only). --rust-debug builds the server with overflow checks on so the
	# edge-range sweep traps arithmetic overflow instead of wrapping it away.
	python3 scripts/regtest.py --no-perftest --no-direct-bench --codegen --language=rust --rust-debug

	- name: Drive the streaming API against the same debug server
	shell: bash
	run: python3 scripts/rust_stream_debug.py

	# Clippy lint gate for the Rust code: the generator tool and the generated crate.
	# The tool enforces #![deny(clippy::pedantic)] on itself, but nothing in CI ever
	# ran clippy, so that gate silently drifted (pedantic errors landed on dev
	# unnoticed). This makes clippy a hard gate. `-D warnings` promotes the remaining
	# warn-level lints (and rustc warnings) to errors too; the generated crate stays
	# green because its scaffolding emits a crate-level #![allow(clippy::all,
	# clippy::pedantic)] — its machine output is verified bit-exact against the C
	# reference, not linted (rewriting emitted comparisons would change NaN semantics).
	# That "(and rustc warnings)" is load-bearing for one lint on purpose: the crate
	# sets #![warn(missing_docs)] (#179 D7), and `-D warnings` here is the only thing
	# that turns an undocumented public item into a failure rather than a warning
	# nobody reads.
	# Rust-only and self-contained: no C build, no JDK/.NET, and intentionally NOT
	# gated on the C hero 'test' job, so an unrelated dist failure can't skip it.
	#
	# This job also RUNS the generator's own unit tests (`cargo test`). Clippy with
	# --all-targets only compiles the test target; it never executes it, so
	# runtime assertions in the generator's tests/*_suite.rs (e.g. the STOCH temp-buffer copy
	# shape) went unchecked and a stale one drifted onto dev unnoticed. Running them
	# here closes that gap at nightly cadence (not per-commit).
	clippy:
	name: Rust clippy + generator tests
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	# Rust is preinstalled on ubuntu-latest; add the component defensively
	# (idempotent — a no-op when clippy is already present).
	- name: Ensure clippy component
	shell: bash
	run: rustup component add clippy

	- name: Clippy the generator tool
	shell: bash
	run: cargo clippy --all-targets --manifest-path ta_codegen/generator/Cargo.toml -- -D warnings

	- name: Clippy the generated crate
	shell: bash
	run: cargo clippy --all-targets --manifest-path ta_codegen/output/rust/Cargo.toml -- -D warnings

	# Rustdoc's own lints are a separate set that clippy never reaches: an
	# unresolved intra-doc link is `rustdoc::broken_intra_doc_links`, and only
	# rustdoc raises it. docs.rs is the first place rustdoc runs on a release,
	# and a broken link renders there as literal text (#179 E4). The registry
	# alone carries 176 generated `[Core::<F>]` links, so this is the step that
	# keeps them resolving.
	- name: Rustdoc the generated crate
	shell: bash
	env:
	RUSTDOCFLAGS: "-D warnings"
	run: cargo doc --no-deps -p ta-lib --manifest-path ta_codegen/output/rust/Cargo.toml

	# --no-fail-fast: cargo runs the test binaries in sequence and stops at
	# the first that fails, so one red suite hid the other eleven -- a real
	# second failure went unseen that way while landing #238.
	- name: Test the generator tool
	shell: bash
	run: cargo test --no-fail-fast --manifest-path ta_codegen/generator/Cargo.toml

	# The generated per-function rustdoc examples are runnable doctests, but
	# nothing executed them: clippy --all-targets does not build doctests. They
	# are the crate's public-facing documentation — run them (issue #136).
	- name: Run the generated crate's doctests
	shell: bash
	run: cargo test --doc -p ta-lib --manifest-path ta_codegen/output/rust/Cargo.toml

	# This step is the ONLY thing anywhere that EXECUTES the crate's
	# hand-written test modules (templates/rust/, copied in by `generate`):
	# types.rs's Core/CoreBuilder API tests (#144), and the four #[cfg(test)]
	# ones -- scratch_election (#146), stream_finite, stream_out_range (#241)
	# and div_zero (#249). clippy --all-targets only COMPILES the test target
	# and --doc skips it, so narrowing this step silently retires all five.
	#
	# `--tests`, not `--lib`: the latter excludes tests/, which is where
	# nullable_outputs.rs (#262) lives. The generated phantom-I/O sweep was
	# there too until #265 moved it in-crate to probe `<N>_Impl`; `--tests`
	# still covers it, because it selects the lib's own unittest binary as
	# well.
	- name: Run the generated crate's unit and integration tests
	shell: bash
	run: cargo test --tests -p ta-lib --manifest-path ta_codegen/output/rust/Cargo.toml

	# Every step above compiles the crate IN the workspace, where each file on
	# disk is reachable whether or not it would ship. The `.crate` tarball is a
	# different artifact -- cargo builds its file list from git, filtered by the
	# manifest's include/exclude and by every .gitignore above it -- and nothing
	# here had ever built one (#179 E1). So a file could be present, compiled,
	# linted, documented and tested and still be absent from what crates.io
	# receives, with a published release that does not build as the first sign.
	#
	# The script packages the pair (the library pins an =0.1.2 dispatch that is
	# not on crates.io yet, so it cannot resolve alone), which makes cargo
	# verification-BUILD each crate from its own unpacked tarball, and then
	# checks the tarball contents against `git ls-files` -- derived, so adding a
	# function needs no edit here, and exact, so losing one fails. Measured
	# against the four steps above by excluding LICENSE from the manifest: all
	# four stay green (clippy, rustdoc, 534 doctests, the --tests suites) and
	# only this step goes red. See scripts/rust_package_check.py.
	#
	# No --allow-dirty on purpose: this runs on a fresh checkout, so an
	# unexpectedly dirty tree is itself something to hear about. No
	# setup-python either -- the script is stdlib-only and ubuntu-latest ships
	# python3, so this job stays the self-contained Rust-only one it is.
	- name: Package the publishable crates and check what ships
	shell: bash
	run: python3 scripts/rust_package_check.py

	# Every Rust step elsewhere in this workflow runs on ubuntu, so the crate
	# has never compiled on Windows or macOS (#179 E6). `--lib --doc` together
	# is rejected by cargo, hence two `cargo test` invocations.
	rust-other-os:
	name: Rust on ${{ matrix.os }}
	runs-on: ${{ matrix.os }}
	timeout-minutes: 45

	strategy:
	fail-fast: false
	matrix:
	os: [windows-2022, macos-latest]

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	- name: Refuse a matrix that adds no operating system
	shell: bash
	run: \|
	os='${{ matrix.os }}'
	if [ -z "$os" ]; then
	echo "::error::This job ran with an empty 'os', so it compiled nothing. The point of the job is to compile the crate somewhere the other Rust jobs do not — restore the matrix in .github/workflows/dev-nightly-tests.yml."
	exit 1
	fi
	case "$os" in
	ubuntu*)
	echo "::error::'$os' is the operating system every other Rust job in this workflow already uses, so this leg would duplicate them instead of covering anything (#179 E6). Name a non-Linux runner, or delete the job rather than leaving it green and vacuous."
	exit 1
	;;
	esac
	echo "Covering $os"

	- name: Compile the publishable crates
	shell: bash
	run: cargo check -p ta-lib -p ta-lib-dispatch --manifest-path ta_codegen/output/rust/Cargo.toml

	- name: Run the crate's unit tests
	shell: bash
	run: cargo test --lib -p ta-lib --manifest-path ta_codegen/output/rust/Cargo.toml

	- name: Run the crate's doctests
	shell: bash
	run: cargo test --doc -p ta-lib --manifest-path ta_codegen/output/rust/Cargo.toml

	# The two publishable crates declare `rust-version = "1.86"`, and nothing
	# else here compiles them at it -- every other Rust step runs on the pinned
	# toolchain from rust-toolchain.toml (1.97.0), so a post-1.86 API reaching
	# the generated crate is green everywhere and breaks only for a consumer on
	# an older toolchain, after publish, in a version that cannot be edited.
	#
	# Nightly, not the PR gate: pr-codegen-gate.yml stays fast and lightweight
	# by design (#298 discussion), and an extra toolchain install of unmeasured
	# download time does not belong in every contributor's merge-blocking path.
	# Nightly is also strictly more coverage, not less -- dev takes direct
	# pushes that `on: pull_request` never sees, so this is the only copy that
	# ever runs against those, the same reasoning regen-check and clippy above
	# already rely on.
	#
	# The floor is READ from the manifests, not written here: bumping it in
	# ta_codegen/generator/src/main.rs moves this gate with it, and the two
	# reads are also the job's non-vacuity guard -- a missing or disagreeing
	# rust-version fails before any toolchain is installed.
	#
	# Rust-only and self-contained, so like clippy above it is deliberately NOT
	# gated on the C hero 'test' job.
	msrv:
	name: Rust MSRV floor
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	- name: Read the declared floor from the two publishable manifests
	id: floor
	shell: bash
	run: \|
	read_msrv() { sed -n 's/^rust-version = "$[^"]$"./\1/p' "$1" \| head -n 1; }
	lib=$(read_msrv ta_codegen/output/rust/library/Cargo.toml)
	dispatch=$(read_msrv ta_codegen/output/rust/dispatch/Cargo.toml)
	if [ -z "$lib" ] \|\| [ -z "$dispatch" ]; then
	echo "::error::A publishable crate declares no rust-version (ta-lib='$lib', ta-lib-dispatch='$dispatch'), so this job has no floor to compile against. Both manifests are generated — restore the literal in ta_codegen/generator/src/main.rs and re-run 'scripts/build.py generate'."
	exit 1
	fi
	if [ "$lib" != "$dispatch" ]; then
	echo "::error::The publishable crates declare different floors (ta-lib='$lib', ta-lib-dispatch='$dispatch'). ta-lib pins ta-lib-dispatch with '=', so the two ship as a pair and only one floor is meaningful. Both literals live in ta_codegen/generator/src/main.rs."
	exit 1
	fi
	echo "Declared MSRV: $lib"
	echo "msrv=$lib" >> "$GITHUB_OUTPUT"

	# rustup is preinstalled on ubuntu-latest. `+<version>` overrides
	# rust-toolchain.toml for this one invocation, so the pin stays the
	# default for every other step and job.
	- name: Install the declared floor toolchain
	shell: bash
	run: rustup toolchain install "${{ steps.floor.outputs.msrv }}" --profile minimal --no-self-update

	- name: Compile the publishable crates on the declared floor
	shell: bash
	run: \|
	if ! cargo "+${{ steps.floor.outputs.msrv }}" check -p ta-lib -p ta-lib-dispatch --all-targets --manifest-path ta_codegen/output/rust/Cargo.toml; then
	echo "::error::The publishable crates do not compile on their own declared MSRV (${{ steps.floor.outputs.msrv }}). Either the code reached for something newer -- a 'use of unstable library feature' or 'stable since 1.NN' note names it -- in which case fix the generator that emits it, or the floor genuinely has to rise, in which case raise rust-version in ta_codegen/generator/src/main.rs (both crates) and say why in the commit. Reproduce locally: rustup toolchain install ${{ steps.floor.outputs.msrv }} && cargo +${{ steps.floor.outputs.msrv }} check -p ta-lib -p ta-lib-dispatch --all-targets --manifest-path ta_codegen/output/rust/Cargo.toml"
	exit 1
	fi

	# Every Rust step in this tree compiles for the runner's own host triple,
	# x86_64-unknown-linux-gnu, and that is the only machine shape the crate has
	# ever been compiled in (#179 E3). Two assumptions rustc can see are invisible
	# from there: a pointer width of 64 -- the crate indexes with `usize` end to
	# end -- and an x86 ISA. Neither has a symptom here; the first sign is a
	# compile error in a consumer's build, on a platform nobody in this repo
	# builds for, in a published version that cannot be edited.
	#
	# Measured on this tree, by injecting each defect into the generated crate and
	# re-running all three legs:
	#
	# injected into library/src/lib.rs host i686 aarch64
	# const _: () = assert!(size_of::<usize>() == 8) green RED green
	# use core::arch::x86_64::_mm_pause; green RED RED
	#
	# So i686 is the leg that does the work: it is the only one of the three that
	# sees a pointer-width assumption. aarch64 earns its place on a different
	# argument -- it is a platform consumers actually run on (Apple Silicon,
	# Graviton), not a defect class i686 misses. No defect was found that aarch64
	# catches and i686 does not.
	#
	# `check`, not `test`: running the crate on either target needs an emulator and
	# a cross linker, and this job deliberately buys neither. A 32-bit arithmetic
	# overflow that only appears at run time is therefore NOT covered here -- this
	# gate is about what the compiler can see.
	#
	# Nightly, not the PR gate, for the same reason as msrv above: two rust-std
	# downloads of unmeasured time do not belong in every contributor's
	# merge-blocking path. Rust-only and self-contained, so also not gated on the
	# C hero 'test' job.
	cross-target:
	name: Rust cross-target check
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	# The list is the job's non-vacuity guard. An empty list, or one that named
	# the host again, would leave every step below green while compiling
	# nothing the other Rust jobs do not already compile -- so both are a hard
	# failure here rather than a silently weaker gate.
	- name: Install the cross targets
	id: targets
	shell: bash
	run: \|
	host=$(rustc -vV \| sed -n 's/^host: //p')
	targets="i686-unknown-linux-gnu aarch64-unknown-linux-gnu"
	if [ -z "${targets// /}" ]; then
	echo "::error::No cross targets are listed, so this job would pass without compiling anything the host jobs do not. Restore the list in .github/workflows/dev-nightly-tests.yml or delete the job."
	exit 1
	fi
	for t in $targets; do
	if [ "$t" = "$host" ]; then
	echo "::error::Cross target '$t' is the runner's own host triple, so that leg re-checks what every other Rust job here already checks. Name a target that differs from the host, or drop it from the list."
	exit 1
	fi
	done
	echo "Host: $host"
	echo "Cross targets: $targets"
	rustup target add $targets
	echo "targets=$targets" >> "$GITHUB_OUTPUT"

	- name: Compile the publishable crates for each cross target
	shell: bash
	run: \|
	for t in ${{ steps.targets.outputs.targets }}; do
	echo "--- $t ---"
	if ! cargo check -p ta-lib -p ta-lib-dispatch --all-targets --target "$t" --manifest-path ta_codegen/output/rust/Cargo.toml; then
	echo "::error::The publishable crates do not compile for $t. The crate is pure Rust with no platform-specific code, so this is a portability assumption that reached the generated output -- a pointer width (anything that assumes usize is 64 bits), an x86-only intrinsic, or a type too large for a 32-bit address space. Fix the generator that emits it in ta_codegen/generator/ and re-run 'scripts/build.py generate'; do not edit ta_codegen/output/. Reproduce locally: rustup target add $t && cargo check -p ta-lib -p ta-lib-dispatch --all-targets --target $t --manifest-path ta_codegen/output/rust/Cargo.toml"
	exit 1
	fi
	done

	# Bit-exact regression gate vs the last release (v0.6.4): diff every function
	# at period>=2, fail on any real divergence. C-only, gated on 'test'.
	# fetch-depth: 0 so the v0.6.4 tag is available. See ta_regtest/CLAUDE.md.
	fuzz-vs-064:
	name: C Latest vs released v0.6.4
	needs: test
	# Run after 'test' but don't let an unrelated failure (e.g. the Windows leg)
	# silently skip this release gate — it has its own checkout+build.
	if: ${{ !cancelled() }}
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev
	fetch-depth: 0 # ta_064_serve builds from the released v0.6.4 tag

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	- name: Fuzz current library vs frozen v0.6.4
	shell: bash
	run: python3 scripts/build.py fuzz-064

	# Native arm64 Linux — the ONE platform where both halves of issue #150 are live
	# at once, and the only reason that issue reached a user instead of CI.
	#
	# (1) Baseline FMA. aarch64 always has it, so any differential oracle built
	# without -ffp-contract=off fuses a*b+c inside the REFERENCE, and the suite
	# then measures a compiler difference rather than a code difference (549
	# false failures across 9 functions, reported on an M5 Max). On x86-64 the
	# flag is inert — no baseline FMA — which is why every job above is blind to
	# this class. macos-latest is arm64, but it runs only the plain C suite; no
	# FMA-baseline runner has ever executed an oracle suite.
	# (2) GCC. fuzz_data.h — the seeded INPUT generator both sides of --fuzz-064
	# compile — defends itself with `#pragma STDC FP_CONTRACT OFF`, which clang
	# honours and GCC ignores outright (GCC also contracts ACROSS statements, so
	# its second defence, splitting multiply from add, goes with it). Only GCC on
	# an FMA-baseline target can therefore make the two sides generate DIFFERENT
	# inputs from the same (shape, seed, n). The reporter could not test this
	# half: macOS /usr/bin/gcc is an Apple clang shim.
	#
	# Not gated on 'test': that job is x86-64 + Windows, so an unrelated dist failure
	# there must never skip this platform's only coverage.
	arm64-linux:
	name: arm64 Linux (GCC, baseline FMA — oracle contraction gate)
	runs-on: ubuntu-24.04-arm
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev
	fetch-depth: 0 # both oracles build from pinned tags (v0.6.4, reference-pre-cutover)

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	# The arm64 images carry a smaller toolset than their x86-64 counterparts,
	# so nothing here is assumed preinstalled the way ubuntu-latest allows.
	- name: Install build tooling
	uses: ./.github/actions/apt-install
	with:
	packages: build-essential cmake

	- name: Install Rust
	shell: bash
	run: \|
	# rust-toolchain.toml pins the version; --default-toolchain none lets
	# rustup honour the pin on first use instead of fetching stable twice.
	if ! command -v cargo >/dev/null 2>&1; then
	curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
	\| sh -s -- -y --default-toolchain none
	echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
	fi

	# First execution of the C suite on arm64 Linux + real GCC anywhere in CI.
	- name: Run the C regression suite
	shell: bash
	run: \|
	python3 scripts/build.py ta_regtest
	cd bin && ./ta_regtest

	# Half (1) and half (2): --fuzz-064 builds the frozen v0.6.4 oracle AND the
	# 064 transport that compiles fuzz_data.h, so it exercises both the cmake
	# sites and the raw-cc site in one run.
	- name: Fuzz current library vs frozen v0.6.4
	shell: bash
	run: python3 scripts/build.py fuzz-064

	# build.py grew the same per-language prerequisite narrowing regtest.py has
	# had since #150, so it needs the same live regression test: on this runner
	# (no JDK, no .NET) the target must build the C and Rust servers without
	# demanding a toolchain it will never invoke. It fails loudly if the
	# prerequisite set ever goes back to being derived from the raw filter.
	- name: build.py honours --language on a runner with no JDK and no .NET
	shell: bash
	run: python3 scripts/build.py servers --language=c,rust

	# The ref differential sweep — the suite that silently truncated on arm64
	# (131 variants/21 functions when it aborted, 613/74 once the oracle stopped
	# fusing). --language=c,rust is also the invocation that was unreachable
	# through the CLI until the prerequisite check became per-language, so this
	# is its live regression test on a runner with no JDK and no .NET.
	- name: Ref differential sweep (C + Rust)
	shell: bash
	run: python3 scripts/regtest.py --no-perftest --no-direct-bench --codegen --language=c,rust

	# C# on arm64 — the one place the managed FMA contract can diverge and nothing
	# would notice. Math.FusedMultiplyAdd is a JIT intrinsic: on x86-64 it becomes
	# vfmadd only when the CPU has FMA3, while on aarch64 FMADD is baseline, so the
	# two platforms reach fused arithmetic by different routes. Every other C# job
	# runs on x86-64, so without this the aarch64 route is never executed at all.
	#
	# Deliberately a SEPARATE job from arm64-linux rather than adding .NET there:
	# that job's value depends on being a runner with no JDK and no .NET, which is
	# what makes its --language=c,rust steps a live regression test.
	#
	# Scoped to the C# row of --xlang-hash (not the full four-language sweep) to
	# keep the nightly's arm64 cost down — it is the row that answers the FMA
	# question, and it is bitwise against the in-process C library on the same
	# machine, so any aarch64-specific fusion difference fails it.
	arm64-csharp:
	name: arm64 Linux (C# bitwise parity — managed FMA on aarch64)
	runs-on: ubuntu-24.04-arm
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	# The arm64 images carry a smaller toolset than their x86-64 counterparts,
	# so nothing is assumed preinstalled the way ubuntu-latest allows.
	- name: Install build tooling
	uses: ./.github/actions/apt-install
	with:
	packages: build-essential cmake

	- name: Install Rust
	shell: bash
	run: \|
	# rust-toolchain.toml pins the version; --default-toolchain none lets
	# rustup honour the pin on first use instead of fetching stable twice.
	if ! command -v cargo >/dev/null 2>&1; then
	curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
	\| sh -s -- -y --default-toolchain none
	echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
	fi

	- name: Install .NET SDK
	uses: actions/setup-dotnet@v6
	with:
	dotnet-version: "10.0.x"

	# --language=csharp reaches this at all only because build.py's prerequisite
	# check and backend filter became per-language; before that the target
	# hard-required a JDK this runner does not have.
	- name: Bitwise parity, C# vs the in-process C library (aarch64)
	shell: bash
	run: python3 scripts/build.py xlang-hash --language=csharp

	# Cross-language BITWISE parity gate (issue #113): the generated language
	# servers must compute BIT-IDENTICAL outputs to the shipped in-process C library
	# (no tolerance — the seed/hex + hash design routes around the JSON-RPC
	# float-serialization limit that forces --codegen's 1e-6). Rust crosses the
	# boundary with a seed (gen_present); Java and C# with lossless hex inputs
	# (#114). Java AND C# relax their transcendental-using calls to a 1e-9
	# tolerance, for different reasons: Java's fdlibm is not the C libm, and .NET
	# does not guarantee Math.* reaches the platform libm (run 30776189041 hit 25
	# TA_LN mismatches here from a commit clean on aarch64 and on a dev box).
	# Everything else in both, and every Rust call, stays bitwise.
	# cmake+gcc+cargo are preinstalled on ubuntu-latest; the Java server needs the
	# JDK and the managed C# server needs the .NET SDK (both set up below).
	xlang-hash:
	name: xlang matching (C vs Rust + Java + C#, seed/hex+hash)
	needs: test
	if: ${{ !cancelled() }}
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	- name: Install JDK
	uses: ./.github/actions/apt-install
	with:
	packages: openjdk-21-jdk

	- name: Install .NET SDK
	uses: actions/setup-dotnet@v6
	with:
	dotnet-version: "10.0.x"

	- name: Bitwise-diff each language server vs the in-process C library
	shell: bash
	run: python3 scripts/build.py xlang-hash

	# musllinux build gate (PR #96): regression guard that the library still BUILDS
	# on musl with FMA dispatch excluded. target_clones/ifunc is glibc-only (musl has
	# no ifunc; a guard wrongly widened to __linux__ would fail the compile here), so
	# a musl build must emit ZERO ifunc and still compute correctly. Checkout on the
	# ubuntu host (Alpine lacks Node for actions/checkout); build + test in an Alpine
	# container. Pinned to `dev`. For a pre-merge local run, paste the `docker run`
	# block below — it needs only Docker and the repo root as $PWD.
	musl-build:
	name: musllinux build (FMA dispatch correctly excluded)
	needs: test
	if: ${{ !cancelled() }}
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	# Pull before running, with retries. Docker Hub times out on the runners often
	# enough to fail a whole nightly on infrastructure rather than on the library:
	# `docker run` exits 125 having built nothing, which reads exactly like a real
	# musl regression until you open the log. Only the pull is retried — the build
	# and the reference suite below are deterministic and must fail on the first try.
	- name: Pull the musl image (retried; Docker Hub is flaky from Actions)
	shell: bash
	run: \|
	for attempt in 1 2 3 4 5; do
	if docker pull --quiet alpine:3.20; then
	echo "pulled alpine:3.20 on attempt $attempt"
	exit 0
	fi
	delay=$((attempt * 15))
	echo "docker pull failed (attempt $attempt/5); retrying in ${delay}s"
	sleep "$delay"
	done
	echo "::error::could not pull alpine:3.20 after 5 attempts — Docker Hub unreachable"
	exit 1

	- name: Build under musl — must compile, emit no ifunc, and compute correctly
	shell: bash
	run: \|
	docker run --rm -v "$PWD":/src -w /src alpine:3.20 sh -c '
	set -e
	apk add --no-cache build-base cmake >/dev/null
	echo "== musl target: alpine $(cat /etc/alpine-release) =="
	# (1) Must COMPILE. A guard wrongly widened to include musl would
	# hard-error here on the target_clones attribute in the fused .c.
	cmake -S . -B /tmp/bm -DCMAKE_BUILD_TYPE=Release >/dev/null
	cmake --build /tmp/bm -j"$(nproc)" >/dev/null
	SO=$(find /tmp/bm -name "libta-lib.so*" -type f \| head -1)
	echo "DSO: $SO"
	# (2) Guard must have EXCLUDED musl: ZERO ifunc dispatch symbols.
	n=$(readelf -sW "$SO" \| grep -c IFUNC \|\| true); echo "IFUNC symbols in DSO (must be 0 on musl): $n"; test "$n" -eq 0
	# (3) The plain software-fma library must still be correct on musl.
	REG=$(find /tmp/bm -name ta_regtest -type f \| head -1)
	echo "== running C reference suite (software fma on musl) =="
	LD_LIBRARY_PATH="$(dirname "$SO")" "$REG"'

	# Memory-safety gate (#94): build the library + ta_regtest with
	# AddressSanitizer + UBSan and run the full C reference suite, including the
	# parameter-boundary sweep (every optional parameter at min/min+1/default+-1/
	# large across every function). Turns an out-of-bounds / uninitialized read in
	# indicator code (the MACD/TRIX/MAVP/STOCH boundary class) into a hard failure
	# instead of plausible garbage. C-only, GCC, gated on 'test'.
	sanitizers:
	name: ASan/UBSan (C reference suite + boundary sweep)
	needs: test
	if: ${{ !cancelled() }}
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	- name: Build ta_regtest with ASan+UBSan
	shell: bash
	run: python3 scripts/build.py ta_regtest --sanitize

	- name: Run C reference suite + boundary sweep under sanitizers
	shell: bash
	working-directory: cmake-build-asan/bin
	env:
	ASAN_OPTIONS: halt_on_error=1:abort_on_error=1
	UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
	run: ./ta_regtest

	# Memory-safety gate for the STREAMING tier. The `sanitizers` job above runs
	# only the batch C-reference suite; it never calls the generated stream
	# functions (Open/Update/Peek/Close, rings, sub-handles). This builds the C
	# JSON-RPC stream server single-TU with -fsanitize=address,undefined (address
	# bundles LeakSanitizer) and drives stream_verify for every stream-flagged
	# function, closing stdin so the server exits CLEANLY and LeakSanitizer runs at
	# exit (driving it through ta_regtest kills the server, so LSan would stay
	# silent). A leak / OOB / UB on any exercised path fails the job. Only needs
	# gcc + Python (the server compiles from the committed generated sources — no
	# cargo). Allocation-failure branches are guarded by generate-time generator
	# tests instead (they require malloc to fail, which no run exercises). C-only,
	# GCC, gated on 'test'.
	stream-sanitizers:
	name: ASan/UBSan/LSan (streaming API)
	needs: test
	if: ${{ !cancelled() }}
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	- name: Install PyYAML
	shell: bash
	run: pip install pyyaml

	- name: Build sanitized stream server + drive stream_verify (ASan/UBSan/LSan)
	shell: bash
	run: python3 scripts/stream_sanitize.py

	# Conan packaging (#86) was never exercised by CI: build the package
	# recipe end to end (library-only; BUILD_DEV_TOOLS stays off in the
	# recipe). Gated on the C hero job like the other non-hero jobs.
	conan:
	name: Conan package build
	needs: test
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	- name: Build conan package
	shell: bash
	run: \|
	pip install conan
	conan profile detect --force
	conan create . --build=missing

	# Runs LAST (needs: test) so every other job tested the committed,
	# already-generated sources. This regenerates from ta_codegen/input/ and fails
	# if anything differs — i.e. an input edit whose generated output was never
	# re-generated and committed.
	regen-check:
	name: ta_codegen output up-to-date
	needs: test
	runs-on: ubuntu-latest
	timeout-minutes: 45

	steps:
	- name: Checkout dev Branch
	uses: actions/checkout@v6
	with:
	ref: dev

	- name: Install Python
	uses: actions/setup-python@v6
	with:
	python-version: "3.x"

	# One command, the same one every PR runs (pr-codegen-gate.yml) and the
	# same one a developer runs locally: the ta_regtest source lists agree,
	# ta_codegen/input is formatted, and regenerating every backend changes
	# nothing that is committed. It probes one file per generated tier first,
	# so "the tree stayed clean" cannot pass for a file the generator no
	# longer writes at all (#211).
	#
	# This job still earns its place next to the PR gate: dev also takes
	# direct pushes, which no pull_request trigger ever sees.
	- name: Regenerate all backends and verify committed output matches
	shell: bash
	run: \|
	if ! python3 scripts/build.py regen-check; then
	echo "::error::ta_codegen output is out of date. Run 'scripts/build.py generate' and commit the result — see the log above for which files drifted."
	exit 1
	fi

	# Static: it reads the generated `src/ta_func/ta_CDL*.c` as TEXT, so it
	# builds and runs nothing. It sits here rather than on the PR because the
	# PR gate is deliberately a fast, lightweight one; systematic checks live
	# in this run. Sub-second, so it is free next to the regenerate above.
	#
	# Nothing else in the tree can see what it checks: a candlestick's only
	# observable is a 3-valued integer, so a trailing total accumulated over
	# the wrong bars changes the output only where a comparison happens to
	# straddle a threshold — on #229's fold, a one-bar rotation moved 3 of 14
	# functions and left 11 green in every language (#240).
	- name: Candle trailing-total wiring (read AND write side)
	shell: bash
	run: \|
	if ! python3 scripts/build.py check-candle-windows; then
	echo "::error::A candlestick's trailing total is wired to the wrong bar — see the log above."
	exit 1
	fi

Navigation Menu

Uh oh!

dev nightly tests #801

Workflow file

dev nightly tests #801

Uh oh!

Workflow file for this run