Skip to content

Commit e1b371b

Browse files
GordonSmithGordonSmith
committed
chore: add integration tests
Signed-off-by: Gordon Smith <[email protected]>
1 parent 7b10ca8 commit e1b371b

29 files changed

+3217
-21
lines changed

‎.github/workflows/test-pr.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,10 @@ jobs:
8282
run: |
8383
npm run test
8484
85+
- name: Test Bundler Integration
86+
run: |
87+
npm run test-bundlers
88+
8589
- name: Calculate Coverage
8690
if: ${{ matrix.os == 'ubuntu-24.04' && matrix.node == 20 }}
8791
run: |

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# @hpcc-js/wasm - Version 3
22

3-
[![Test PR](https://github.com/hpcc-systems/hpcc-js-wasm/actions/workflows/test-pr.yml/badge.svg)](https://github.com/hpcc-systems/hpcc-js-wasm/actions/workflows/test-pr.yml)
3+
[![Test PR](https://github.com/hpcc-systems/hpcc-js-wasm/actions/workflows/test-pr.yml/badge.svg?branch=release-please--branches--trunk)](https://github.com/hpcc-systems/hpcc-js-wasm/actions/workflows/test-pr.yml)
44
[![release-please](https://github.com/hpcc-systems/hpcc-js-wasm/actions/workflows/release-please.yml/badge.svg)](https://github.com/hpcc-systems/hpcc-js-wasm/actions/workflows/release-please.yml)
55
[![Coverage Status](https://coveralls.io/repos/github/hpcc-systems/hpcc-js-wasm/badge.svg?branch=BUMP_VERSIONS)](https://coveralls.io/github/GordonSmith/hpcc-js-wasm?branch=BUMP_VERSIONS)
66

‎docs/.vitepress/config.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ export default {
22
title: '@hpcc-js/wasm',
33
description: 'HPCC Systems Wasm Libraries',
44
base: '/hpcc-js-wasm/',
5+
srcExclude: ['**/tests/**'],
56

67
themeConfig: {
78
repo: "hpcc-systems/hpcc-js-wasm",

‎docs/bundler-integration-tests.md‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
# Bundler Integration Tests
2+
3+
## Overview
4+
5+
This test suite verifies that the built `@hpcc-js/wasm-*` packages can be properly consumed by popular JavaScript bundlers. The tests ensure that WebAssembly modules load correctly and function as expected when bundled with different build tools.
6+
7+
## Architecture
8+
9+
```
10+
tests/bundlers/
11+
├── src/
12+
│ ├── test-utils.js # Shared test functions
13+
│ ├── webpack-test.js # Webpack entry point
14+
│ ├── esbuild-test.js # ESBuild entry point
15+
│ └── rollup-test.js # Rollup entry point
16+
├── webpack.config.js # Webpack configuration
17+
├── rollup.config.js # Rollup configuration
18+
├── vitest.config.js # Vitest test configuration
19+
├── bundlers.test.js # Main test suite
20+
├── package.json # Test dependencies
21+
└── README.md # Documentation
22+
```
23+
24+
## Tested Combinations
25+
26+
### Bundlers
27+
- **Webpack 5** with WebAssembly async support
28+
- **ESBuild** with Node.js target
29+
- **Rollup.js** with ES module output
30+
31+
### Packages
32+
- `@hpcc-js/wasm-base91` - Base91 encoding/decoding
33+
- `@hpcc-js/wasm-expat` - XML parsing with Expat
34+
- `@hpcc-js/wasm-graphviz` - Graphviz DOT rendering
35+
- `@hpcc-js/wasm-duckdb` - DuckDB in-process SQL OLAP database
36+
- `@hpcc-js/wasm-llama` - Large Language Model inference
37+
- `@hpcc-js/wasm-zstd` - Zstandard compression
38+
39+
## Running Tests
40+
41+
### From Root Directory
42+
43+
```bash
44+
# Install all dependencies including bundler test deps
45+
npm run install-build-deps
46+
47+
# Run all bundler tests via vitest
48+
npm run test-bundlers
49+
50+
# Run bundler tests via shell script (more verbose)
51+
npm run test-bundlers-script
52+
53+
# Run specific bundler tests
54+
npm run test-bundlers-webpack
55+
npm run test-bundlers-esbuild
56+
npm run test-bundlers-rollup
57+
58+
# Run all tests (regular + bundler)
59+
npm run test-all
60+
```
61+
62+
### From Bundlers Directory
63+
64+
```bash
65+
cd tests/bundlers
66+
67+
# Install dependencies
68+
npm install
69+
70+
# Run vitest integration tests
71+
npm test
72+
73+
# Run specific bundler manually
74+
npm run test:webpack
75+
npm run test:esbuild
76+
npm run test:rollup
77+
```
78+
79+
## Test Process
80+
81+
For each bundler, the test:
82+
83+
1. **Bundles** the test code with the bundler's configuration
84+
2. **Executes** the bundled output in Node.js
85+
3. **Verifies** that each WASM package loads successfully
86+
4. **Tests** core functionality of each package
87+
5. **Confirms** no runtime errors occur
88+
89+
## What Gets Tested
90+
91+
### Package Loading
92+
- Correct module resolution
93+
- WebAssembly module loading
94+
- Proper initialization sequencing
95+
96+
### Functionality
97+
- **Base91**: Encode/decode string data
98+
- **Expat**: Parse XML documents with callbacks
99+
- **Graphviz**: Render DOT notation to SVG
100+
- **DuckDB**: Execute SQL queries on in-memory database
101+
- **Llama**: Load models and verify functionality
102+
- **Zstd**: Compress/decompress binary data
103+
104+
### Integration Points
105+
- ES module imports
106+
- WebAssembly instantiation
107+
- Async module initialization
108+
- Cross-package compatibility
109+
110+
## Bundler Configurations
111+
112+
### Webpack
113+
- WebAssembly async experiments enabled
114+
- Node.js target for testing
115+
- Built-in modules externalized
116+
- WASM files handled as async modules
117+
118+
### ESBuild
119+
- Node.js platform target
120+
- Bundle mode enabled
121+
- External Node.js modules
122+
123+
### Rollup
124+
- ES module output format
125+
- Node.js built-ins externalized
126+
- Module resolution via plugins
127+
128+
## CI Integration
129+
130+
The tests are integrated into the GitHub Actions workflow (`test-pr.yml`) and run on:
131+
- Ubuntu, macOS (Intel & ARM)
132+
- Node.js versions 22, 24
133+
- Every pull request and push
134+
135+
## Troubleshooting
136+
137+
### Common Issues
138+
139+
**WebAssembly Loading Errors**
140+
- Ensure WASM files are included in package distributions
141+
- Check bundler WebAssembly support is configured correctly
142+
143+
**Module Resolution Errors**
144+
- Verify workspace dependencies are properly linked
145+
- Check that packages are built before running tests
146+
147+
**Timeout Errors**
148+
- Bundling operations can be slow; tests have extended timeouts
149+
- Large WebAssembly modules may need more time to initialize
150+
151+
### Debugging
152+
153+
Enable verbose output:
154+
```bash
155+
# Use the shell script for detailed output
156+
npm run test-bundlers-script
157+
158+
# Or run bundlers individually
159+
cd tests/bundlers
160+
npm run test:webpack # etc.
161+
```
162+
163+
Check bundled output:
164+
```bash
165+
cd tests/bundlers
166+
npm run build:webpack
167+
node dist/webpack-test.js
168+
```
169+
170+
## Future Enhancements
171+
172+
Potential areas for expansion:
173+
174+
- **More Bundlers**: Parcel, Vite, etc.
175+
- **Browser Testing**: Test bundled code in browsers
176+
- **Tree Shaking**: Verify unused code elimination
177+
- **Code Splitting**: Test dynamic imports and chunks
178+
- **Different Targets**: Browser, Web Worker, Service Worker
179+
- **Package Combinations**: Test using multiple packages together

‎docs/llama/src/llama/README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
[**@hpcc-js/wasm-root**](../../../README.md)
2+
3+
***
4+
5+
# llama/src/llama
6+
7+
## Classes
8+
9+
- [Llama](/llama/src/llama/classes/Llama.md)

‎docs/llama/src/llama/classes/Llama.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
[**@hpcc-js/wasm-root**](../../../../README.md)
2+
3+
***
4+
5+
# Class: Llama
6+
7+
Defined in: [llama/src/llama.ts:27](https://github.com/GordonSmith/hpcc-js-wasm/blob/10467e7b89384555d9ecd41bd4f60b5c63687e4a/packages/llama/src/llama.ts#L27)
8+
9+
The llama WASM library, provides a simplified wrapper around the llama.cpp library.
10+
11+
See [llama.cpp](https://github.com/ggerganov/llama.cpp) for more details.
12+
13+
```ts
14+
import { Llama, WebBlob } from "@hpcc-js/wasm-llama";
15+
16+
let llama = await Llama.load();
17+
const model = "https://huggingface.co/CompendiumLabs/bge-base-en-v1.5-gguf/resolve/main/bge-base-en-v1.5-q4_k_m.gguf";
18+
const webBlob: Blob = await WebBlob.create(new URL(model));
19+
20+
const data: ArrayBuffer = await webBlob.arrayBuffer();
21+
22+
const embeddings = llama.embedding("Hello and Welcome!", new Uint8Array(data));
23+
```
24+
25+
## Methods
26+
27+
### load()
28+
29+
> `static` **load**(): `Promise`\<`Llama`\>
30+
31+
Defined in: [llama/src/llama.ts:41](https://github.com/GordonSmith/hpcc-js-wasm/blob/10467e7b89384555d9ecd41bd4f60b5c63687e4a/packages/llama/src/llama.ts#L41)
32+
33+
Compiles and instantiates the raw wasm.
34+
35+
::: info
36+
In general WebAssembly compilation is disallowed on the main thread if the buffer size is larger than 4KB, hence forcing `load` to be asynchronous;
37+
:::
38+
39+
#### Returns
40+
41+
`Promise`\<`Llama`\>
42+
43+
A promise to an instance of the Llama class.
44+
45+
***
46+
47+
### unload()
48+
49+
> `static` **unload**(): `void`
50+
51+
Defined in: [llama/src/llama.ts:50](https://github.com/GordonSmith/hpcc-js-wasm/blob/10467e7b89384555d9ecd41bd4f60b5c63687e4a/packages/llama/src/llama.ts#L50)
52+
53+
Unloades the compiled wasm instance.
54+
55+
#### Returns
56+
57+
`void`
58+
59+
***
60+
61+
### version()
62+
63+
> **version**(): `string`
64+
65+
Defined in: [llama/src/llama.ts:57](https://github.com/GordonSmith/hpcc-js-wasm/blob/10467e7b89384555d9ecd41bd4f60b5c63687e4a/packages/llama/src/llama.ts#L57)
66+
67+
#### Returns
68+
69+
`string`
70+
71+
The Llama c++ version
72+
73+
***
74+
75+
### embedding()
76+
77+
> **embedding**(`text`, `model`, `format`): `number`[][]
78+
79+
Defined in: [llama/src/llama.ts:69](https://github.com/GordonSmith/hpcc-js-wasm/blob/10467e7b89384555d9ecd41bd4f60b5c63687e4a/packages/llama/src/llama.ts#L69)
80+
81+
Calculates the vector representation of the input text.
82+
83+
#### Parameters
84+
85+
##### text
86+
87+
`string`
88+
89+
The input text.
90+
91+
##### model
92+
93+
`Uint8Array`
94+
95+
The model to use for the embedding.
96+
97+
##### format
98+
99+
`string` = `"array"`
100+
101+
#### Returns
102+
103+
`number`[][]
104+
105+
The embedding of the text using the model.

0 commit comments

Comments
 (0)