Add `tools/provider-tck`: a Java conformance suite for OpenFeature providers

[aepfli]

Part of open-feature/spec#417, the cross-language tracking issue for the provider conformance suite. This issue is the Java half; the language-agnostic artifacts (Gherkin, canonical flag set, control API) are proposed for the spec repo in spec#423.

Summary

A new module, tools/provider-tck, containing the Java implementation of the OpenFeature Provider
TCK: the canonical Gherkin, the Cucumber step definitions, and an abstract JUnit Platform Suite that
owns the entire test lifecycle. Provider authors implement a factory interface and supply a
docker-compose file; everything else belongs to the TCK.

flagd is the first adopter, wrapping the existing flagd-testbed image without modification.

Status: working proof of concept. 15 scenarios, running green against the flagd provider (14
passed, 1 visibly skipped — see Findings).

Why a new module rather than extending the flagd e2e suite

The flagd provider's existing e2e suite (providers/flagd/src/test/.../e2e/) is close to what we
want and is where the design comes from, but it is not reusable as-is:

• Features are selected from the filesystem (@SelectDirectories("test-harness/gherkin")), so every
  consumer would need their own git submodule.
• State.resolverType is a static set by each suite's @BeforeSuite — flagd-specific and
  single-suite-per-JVM.
• Step definitions bind directly to FlagdOptions / FlagdProvider.
• The provider setup step branches on flagd-specific flavours (ssl, socket, syncpayload,
  metadata, forbidden).

The reusable core is the vocabulary and the control-API interaction pattern, which this module
extracts. The flagd suite stays exactly as it is.

Module layout

tools/provider-tck/
├── pom.xml                                   groupId dev.openfeature.contrib.tools, v0.0.1
├── README.md                                 the provider-author guide
└── src/main/
    ├── resources/
    │   ├── features/*.feature                canonical Gherkin, packaged IN THE JAR
    │   ├── flags/canonical-flags.json        the flag set vendors must seed
    │   └── openapi/control-api.yaml          the backend control API contract
    └── java/dev/openfeature/contrib/tools/providertck/
        ├── ProviderTckHarness.java           the SPI — the entire vendor contract
        ├── AbstractProviderTckTest.java      the @Suite base, annotations only
        ├── BackendEndpoint.java              host + mapped-port lookup
        ├── Capability.java                   capability ↔ Gherkin tag
        ├── ControlApiClient.java             JDK HttpClient, implements control-api.yaml
        ├── TckRuntime.java                   suite-scoped: harness discovery + Compose lifecycle
        ├── TckState.java                     Pico-injected scenario state
        └── steps/{Provider,Flag,Event,Context}Steps.java

Structural choices follow tools/flagd-api-testkit, which already solved the same adoption problem
for the flagd-api Evaluator:

• Features live in src/main/resources/features and are selected with
  @SelectClasspathResource("features") — consumers need no submodule.
• Normally test-scoped dependencies are promoted to compile scope so the abstract suite and the
  glue resolve downstream.
• Harness discovery is ServiceLoader, exactly like EvaluatorFactory.

HTTP uses the JDK's java.net.http.HttpClient rather than RestAssured, so adopting the TCK does not
push an HTTP library onto a provider's test classpath.

The adoption surface

public interface ProviderTckHarness {
    File composeFile();                                     // the stack under test
    List<Integer> backendPorts();                           // internal ports to expose
    FeatureProvider createProvider(BackendEndpoint e);      // ports only known after startup
    FeatureProvider createUnavailableProvider();            // for init-failure scenarios

    // conventions, all defaulted
    default Set<Capability> capabilities()        { return EnumSet.allOf(Capability.class); }
    default String backendService()               { return "backend"; }
    default int controlPort()                     { return 8080; }
    default Map<String, List<Integer>> additionalExposedPorts() { return emptyMap(); }
    default String defaultConfig()                { return "default"; }
    default Duration eventTimeout()               { return Duration.ofSeconds(12); }
    default Duration readyTimeout()               { return Duration.ofSeconds(30); }
    default Duration startupTimeout()             { return Duration.ofSeconds(60); }
    default Duration settleTime()                 { return Duration.ofMillis(50); }
}

createProvider is a factory rather than a field because external ports are dynamically mapped and
therefore unknown until the stack is running — which is also why stacks must never pin host ports.

The event timeout is the knob that matters most in practice: a streaming provider sees a
configuration change in milliseconds, a provider polling every 30 seconds needs most of a poll
interval. A hard-coded constant (as in today's flagd suite,
EVENT_TIMEOUT_MS = 12_000) cannot serve both.

The flagd adoption

The whole thing, and the yardstick for whether the abstraction leaks:

• providers/flagd/src/test/resources/tck/docker-compose.yaml — 14 lines, wrapping
  ghcr.io/open-feature/flagd-testbed:v3.8.0 directly. No envoy, no volume mount, and no
  test-harness submodule required for the TCK path.
• providers/flagd/src/test/java/.../e2e/FlagdTckTest.java — ~40 lines: four overrides.
• providers/flagd/src/test/resources/META-INF/services/...ProviderTckHarness — one line.

Placing the test at .../flagd/e2e/FlagdTckTest.java means the existing
<testExclusions>**/e2e/*.java</testExclusions> already excludes it from default builds and the
existing -P e2e profile already enables it. The only POM change to providers/flagd is the
test-scoped dependency.

flagd-testbed is not modified.

Scenario coverage

A representative subset covering each architectural mechanism once, not exhaustive coverage.

Feature	Scenarios
evaluation.feature	boolean/string/integer/float with value + variant + reason; integer and float resolve as distinct types; @object structured value
errors.feature	TYPE_MISMATCH returns the code default and does not throw; @strict-numeric-typing float is not narrowed to integer; FLAG_NOT_FOUND returns the code default and does not throw
lifecycle.feature	READY + PROVIDER_READY; @unavailable init against a dead backend gives ERROR + PROVIDER_ERROR; an errored provider still returns code defaults
events.feature	@configuration-change signalled and applied; @stale lost → STALE + PROVIDER_STALE, restored → READY + PROVIDER_READY

Step vocabulary

Inherited from the flagd test harness wherever it was already provider-neutral, so flagd's existing
features port with a near-zero diff. Only vendor-specific wording changed:

flagd	TCK
Given a stable flagd provider	Given a stable provider
Given a unavailable flagd provider	Given a unavailable provider

Three steps are new, each for a concrete reason:

• When the connection is restored — the flagd harness has only the self-healing
  the connection is lost for {int}s, which cannot express "assert the provider is stale, then
  reconnect": the reconnect races the assertion. Splitting the outage makes the stale→ready
  transition deterministic.
• When the resolved value is remembered / Then the resolved details value should have changed — the control API only requires that /change changes changing-flag's value, not
  which value it changes to. Asserting a delta keeps the scenario vendor-neutral and independent of
  how many times it has run against the same stack.
• Then no exception should have been thrown — makes the "typed evaluation never throws" half
  of the error contract explicit rather than implicit in a step failure.

Findings

Two things surfaced on the first real run, both worth their own follow-ups.

1. The flagd provider silently narrows a float flag to an integer

Evaluating float-flag (0.5) through getIntegerDetails returns 0 with no error code at
all — not TYPE_MISMATCH with the code default. The application sees a plausible value and no
indication anything went wrong.

Handled for now by not declaring Capability.STRICT_NUMERIC_TYPING in FlagdTckTest, so the
scenario is reported as skipped with the reason printed rather than silently passing. The
FlagdTckTest javadoc says plainly that this is a defect to fix, not a design choice, and that the
line should be deleted once it is. Needs a separate issue against the flagd provider.

2. Cucumber parallelism silently breaks the suite

providers/flagd/src/test/resources/junit-platform.properties sets
cucumber.execution.parallel.enabled=true, which the TCK suite inherited simply by being on that
module's classpath. Scenarios then raced each other's control-API calls — one scenario's /start
restarted the backend underneath another's disconnect assertion — and the symptom looked like a
flaky provider, not a broken test.

AbstractProviderTckTest now pins parallel.enabled=false and execution-mode.feature=same_thread
via @ConfigurationParameter, which overrides any junit-platform.properties in the consuming
module. Worth knowing that this trap exists for any provider that runs Cucumber in parallel — which
is why it is enforced in the base class rather than documented in the README.

Verification

Check	Result
mvn --projects tools/provider-tck -P codequality,deploy clean verify	passes — checkstyle, PMD, SpotBugs, Spotless/Palantir, javadoc with failOnWarnings=true
Compiles under maven.compiler.release=11	yes — Testcontainers 2.x targets Java 8 bytecode (options.release.set(8)), so it links fine on the compile classpath
mvn --projects providers/flagd -P e2e test -Dtest=FlagdTckTest	15 scenarios: 14 passed, 1 skipped, 0 failed
Capability gating reports skips visibly	yes — the undeclared @strict-numeric-typing scenario is reported skipped with its reason
Compose stack started exactly once per suite	yes
Existing flagd e2e suites unaffected	verified under -P e2e

Testcontainers is pinned to 2.0.4, matching what providers/flagd already uses, so no two
Testcontainers majors end up on one test classpath.

Repo wiring

Follows the CONTRIBUTING.md "Adding a module" checklist: <module> entry in the root POM, a
release-please-config.json package block, a .release-please-manifest.json entry at 0.0.1,
version.txt, README.md, and a .github/component_owners.yml entry.

Open items

• The spec artifacts should move to open-feature/spec. The features, the control API spec and
  the canonical flag set are language-agnostic contract definitions, not Java artifacts. The POM
  carries a comment describing the migration, which is mechanically identical to what
  flagd-api-testkit already does with its test-harness submodule; consumers see no difference.
• Context passthrough is unverifiable. The TCK builds evaluation contexts but cannot assert one
  reached the backend. That needs an echo operation on the control API. A provider that silently
  drops the context passes today.
• TckRuntime is static, so one TCK suite may run per JVM fork at a time. Adequate for now;
  worth revisiting if a provider wants several transport variants in one build (they can already
  use one Surefire execution each with -Dopenfeature.tck.harness=).
• Not yet ported from the flagd harness: flag metadata scenarios, caching, targeting, hooks.
• flagd in-process mode is not yet covered — it is a copy of FlagdTckTest with a different
  resolver, deliberately left out of the PoC to keep the reviewable surface small.

Pull requests

PR	What
#1830	tools/provider-tck — library, Gherkin, step definitions, containerised base class, flagd adoption for both resolvers
#1837	in-process control path: BackendControl seam, base-class split, in-memory and multi-provider self-tests, Docker-free CI job
#1838	sources the Gherkin, flag set and control API from the open-feature/spec submodule instead of keeping copies here. Depends on spec#423.

Stacked in that order: #1838 → #1837 → #1830 → main.

Two of the "known limitations" above are now addressed by #1837: flagd in-process mode is covered (both resolvers, one suite class each), and the single-suite-per-fork constraint is unchanged but no longer needs a system property, since the executing suite is discovered from the JUnit test plan.

Findings so far

The suite has surfaced two real defects from the outside, which is the argument for having it:

• flagd silently narrows a float flag to an integer — float-flag (0.5) through the integer API returns 0 with no error code, rather than TYPE_MISMATCH with the code default. Reported as a @strict-numeric-typing skip rather than a pass.
• MultiProvider swallows child provider events — it extends EventProvider but never subscribes to its children, so PROVIDER_CONFIGURATION_CHANGED, PROVIDER_ERROR and PROVIDER_STALE from a child never reach the client. Independently reproduced by the TCK; already tracked as open-feature/java-sdk#1882 (gap 1, High).