test: replace tsd with vitest type checking

[outslept]

Summary

Part of the effort to move all your tests to a single test runner. Note, that currently the script runs both type tests and the single existing vitest runtime test together, which is not a big deal now. Renamed files to *.test-d.ts following vitest convention.

---

For us to review and ship your PR efficiently, please perform the following steps:

• Open a bug/issue before writing your code 🧑‍💻. This ensures
  we can discuss the changes and get feedback from everyone that should be involved. If you`re fixing a typo or
  something that`s on fire 🔥 (e.g. incident related), you can skip this step.
• Read the contribution guidelines 📖. This ensures
  your code follows our style guide and passes our tests.
• Update or add tests (if any source code was changed or added) 🧪
• Update or add documentation (if features were changed or added) 📝
• Make sure the status checks below are successful ✅

A picture of a cute animal (not mandatory, but encouraged)

[mrazauskas]

To draw your attention, the type testing feature is marked experimental in Vitest. I can’t find any further detail on that. Looking few the issues, these might explain:

• using references in TSConfig is not supported (can be a problem for a monorepo), vitest typecheck does not work when TypeScript references are used with noEmit vitest-dev/vitest#3752
• exactOptionalPropertyTypes: true is not taken into account, false equality with exactOptionalPropertyTypes mmkal/expect-type#128

I understand it is tempting to use the same tool for everything, but note that tsd does not have these issues. In a way this change is potentially introducing regressions.

---

Further more, assignability matchers do not exist in the Vitest implementation. A test like:

build/packages/build/test-d/netlify_plugin.ts

Line 11 in 2ea5a35

expectAssignable<OnEnd>(noop)

gets turned into:

expectTypeOf(noop).toExtend<OnEnd>()

The problem here is readability. Is the type of noop under test? And .toExtend<OnEnd>() is the condition to check?

Seems like the idea behind the tsd test was roughly: let x: OnEnd = noop. With .toExtend() that is not clear anymore.

At first glance the tsd test could look correct, but in fact actual and expected are swapped in its assertions:

build/packages/build/test-d/netlify_plugin.ts

Line 21 in 2ea5a35

expectType<Error | undefined>(error)

Here error and in the above assertion noop are under the test.

---

.not.toBeCallableWith() is missing and expectError() also does not exist either. The following:

build/packages/build/test-d/options/netlify_plugin_run_util.ts

Line 8 in 2ea5a35

expectError(run('command', ['arg'], { unknownOption: false }))

is rewritten to:

// @ts-expect-error - testing invalid
expectTypeOf<RunFunction>().toBeCallableWith('command', ['arg'], { unknownOption: false })

Note that // @ts-expect-error hides any error. Typos, missing imports, anything. Copy paste these lines into an empty file and that is a passing test already.

---

In the other hand, tsd haș its own problems. For instance:

• in some cases test are passing with not and without: test pass with both expectType and expectNotType, pls help me understand tsdjs/tsd#141
• it does not list test files
• the install size is 39MB

What is good about tsd: it is a domain specific tool solving a particular problem.

---

As an alternative, I would like to suggest trying out TSTyche (https://tstyche.org). It is a type testing tool created by me. TSTyche can do more than any other similar tool (and does not have issues mentioned above):

• handles any projects of any complexity (takes into account references)
• has assignability matchers: expect<OnEnd>().type.toBeAssignableWith(noop)
• has .not.toBeCallableWith()
• checks error messages suppressed by // @ts-expect-error
• can run tests agains specific TypeScript version or range of version: tstyche --target '>=5.4'
• the install size is 260kB

To say it short, TSTyche is simply the same old “do one thing and do it well” type of toll.

This is rather lengthy comment. Thanks for reading.

[outslept]

@mrazauskas Thanks for your detailed review. Just want to clarify that the final decision on tooling and codebase direction is up to the Netlify team - they can reject this PR and stick with tsd, or consider TSTyche as you suggested.

I'll briefly address the technical concerns below. For broader discussions about alternative tooling approaches, we might want to continue that conversation elsewhere to keep this PR focused on the specific changes.

• Experimental status

There's conflicting information in Vitest's docs. The testing-types guide presents it as standard, while the features page has an experimental badge. Looking at the implementation, it's essentially a wrapper around expect-type utilities with TypeScript compiler integration - shouldn't be particularly concerning for most consumers.

• TypeScript references not supported

Not applicable. They don't use project references for that package - the base config has "composite": true commented out, and the package config has no "references" section

• exactOptionalPropertyTypes: true not taken into account

Root tsconfig.base.json has "exactOptionalPropertyTypes": false, and the package config inherits this via "extends": "../../tsconfig.base.json", so this limitation doesn't affect the package.

• Missing assignability matchers

👍🏻

• Readability concerns

👍🏻

• Missing functionality (.not.toBeCallableWith(), expectError())

👍🏻

Regarding code review, I appreciate the feedback and will iterate on these points.

[mrazauskas]

Thanks for the response. What I try to explain is the context and alternatives. The PR description did not included this information. (Of course, there isn’t a single solution that works for each and every project.)

Documentation might be inconstant, but the experimental message is logged each time the tests run via CLI:

I agree, that is not a problem right now. Same with support for references or exactOptionalPropertyTypes. Although is that future proof? Hard to say. But this is clear: the current setup does not have these limitations.

[mrazauskas]

Suggested change

	// @ts-expect-error - TypeScript has no negative-call assertion; this error is the test
	// @ts-expect-error - 'expect-type' has no negative-call assertion; this error is the test

[mrazauskas]

By the way, adding some obviously failing assertion:

expectTypeOf<string>().toEqualTypeOf<number>()

or a line with an error:

const a: string = 123

to any of the test files?

I see errors in the editor, but Vitest lists all the tests files as passing. Seems like the tests are always passing despite obvious failures, or?

[outslept]

@mrazauskas You might be mixing up Vitest’s type tests with tsd. In Vitest, type tests are statically checked (not executed) and only run when you enable typechecking—use vitest --typecheck and keep them in *.test-d.ts (or configure typecheck.include). Under the hood Vitest calls tsc --noEmit and treats type errors as test failures; without typechecking, esbuild strips types and those tests appear to “pass.”

[mrazauskas]

treats type errors as test failures

Yes, that what I would expect to see. But.. Nothing.

What I do wrong? I checked out this branch, installed dependencies, added failing lines to one of the type test files, cd packages/build and npm run test:types -- --run. The result:

[outslept]

It's the tsconfig. build's package tsconfig uses files/include that didn’t cover test-d, so vitest --typecheck ran but tsc never saw the failing assertions. This could be fixed by adding a dedicated tsconfig for type tests and pointing Vitest to it. It's a leftover from the previous setup. Good catch

[mrazauskas]

Don’t you think this is Vitest’s responsibility to notice this problem? Listing test files as passing without checking them, sounds rather misleading.