Skip to content

Commit a6c8cab

Browse files
GordonSmithGordonSmith
authored
Merge pull request #323 from hpcc-systems/copilot/fix-322
Create comprehensive GitHub Copilot instructions for WASM monorepo development
2 parents 4f3f472 + 9e0af0c commit a6c8cab

File tree

1 file changed

+324
-0
lines changed

1 file changed

+324
-0
lines changed

‎.github/copilot-instructions.md‎

Lines changed: 324 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,324 @@
1+
# HPCC-JS-WASM Copilot Instructions
2+
3+
**ALWAYS follow these instructions first**. Only fallback to additional search and context gathering if the information here is incomplete or found to be in error.
4+
5+
## Repository Overview
6+
7+
This is a WebAssembly (WASM) monorepo providing JavaScript/TypeScript bindings for C++ libraries:
8+
- @hpcc-js/wasm-base91 - Base91 encoding/decoding
9+
- @hpcc-js/wasm-duckdb - DuckDB embedded database
10+
- @hpcc-js/wasm-expat - Expat XML parser
11+
- @hpcc-js/wasm-graphviz - Graphviz graph visualization
12+
- @hpcc-js/wasm-graphviz-cli - Graphviz CLI tool
13+
- @hpcc-js/wasm-llama - Llama.cpp AI models
14+
- @hpcc-js/wasm-zstd - Zstandard compression
15+
- @hpcc-js/wasm - Meta package for backward compatibility
16+
17+
## Critical Build Requirements
18+
19+
**NEVER CANCEL BUILD COMMANDS** - Full builds can take 45+ minutes. ALWAYS set timeouts to 60+ minutes for any build command.
20+
21+
### Initial Setup Commands
22+
```bash
23+
# 1. Install dependencies (2-3 minutes)
24+
npm ci
25+
26+
# 2. Install build dependencies (20-30 minutes - NEVER CANCEL)
27+
# Includes: emsdk install (~5 min), vcpkg install (~15-20 min), playwright (~5-10 min)
28+
npm run install-build-deps
29+
30+
# 3. Build C++ to WASM (30-45 minutes - NEVER CANCEL)
31+
# CRITICAL: Set timeout to 3600+ seconds (60+ minutes)
32+
npm run build-cpp
33+
34+
# 4. Build TypeScript packages (5-10 minutes)
35+
npm run build-ws
36+
37+
# 5. Full build command (45+ minutes total - NEVER CANCEL)
38+
# Equivalent to: build-cpp && build-ws
39+
npm run build
40+
```
41+
42+
### Measured Timing Examples
43+
From actual runs in clean environment:
44+
- `npm ci`: 44 seconds
45+
- `npm run lint`: 1-2 minutes (after TypeScript build)
46+
- `npm run install-emsdk`: 5-10 minutes (downloading + setup)
47+
- `npm run install-vcpkg`: 15-25 minutes (depends on network)
48+
- `npx playwright install`: 10-20 minutes (browser downloads)
49+
- `npm run build-cpp`: 30-60 minutes (C++ compilation to WASM)
50+
- `npm run test`: 15-30 minutes (includes browser and node tests)
51+
52+
### Quick Development (TypeScript Only)
53+
```bash
54+
# For TypeScript-only changes when WASM files exist
55+
npm ci # 2-3 minutes
56+
npm run build-ws # 5-10 minutes
57+
npm run lint # 2-3 minutes
58+
```
59+
60+
## Working Without Full Build
61+
62+
### What Works with npm ci + npm run lint Only
63+
- Code style checking and linting
64+
- TypeScript type checking (tsc --noEmit)
65+
- Documentation viewing and editing
66+
- Package.json script modification
67+
- Configuration file changes
68+
69+
### What Requires npm run build-ws
70+
- TypeScript compilation to JavaScript
71+
- Package distribution files (dist/ directories)
72+
- Import/export resolution testing
73+
- ESLint with compiled output
74+
75+
### What Requires Full WASM Build (npm run build-cpp + build-ws)
76+
- Running any package functionality
77+
- All tests (browser and node)
78+
- Bundle testing (webpack, rollup, esbuild)
79+
- Actual library usage and validation
80+
- Performance testing
81+
82+
**WARNING**: TypeScript-only builds WILL FAIL if WASM files don't exist. You'll see errors like:
83+
```
84+
✘ [ERROR] Could not resolve "../../../build/packages/base91/src-cpp/base91lib.wasm"
85+
✘ [ERROR] Could not resolve "../../../build/packages/graphviz/src-cpp/graphvizlib.wasm"
86+
```
87+
88+
### DuckDB Exception
89+
The DuckDB package is special:
90+
```bash
91+
cd packages/duckdb
92+
npm run pack-duckdb # Uses pre-built WASM from @duckdb/duckdb-wasm
93+
npm run build # Can complete without C++ compilation
94+
```
95+
This works because DuckDB uses pre-compiled WASM binaries rather than building from C++ source.
96+
```
97+
ERROR: Could not resolve "../../../build/packages/graphviz/src-cpp/graphvizlib.wasm"
98+
```
99+
100+
## Testing
101+
102+
### Prerequisites
103+
```bash
104+
# Install playwright browsers (10-15 minutes - NEVER CANCEL)
105+
npx playwright install --with-deps
106+
```
107+
108+
### Test Commands
109+
```bash
110+
# Run all tests (15-20 minutes - NEVER CANCEL)
111+
npm run test
112+
113+
# Run specific package tests
114+
cd packages/graphviz && npm test
115+
116+
# Run browser tests only (10-15 minutes)
117+
npm run test-browser
118+
119+
# Run node tests only (5-10 minutes)
120+
npm run test-node
121+
122+
# Run bundler integration tests (5-10 minutes)
123+
npm run test-bundlers
124+
```
125+
126+
**CRITICAL**: Tests require full build completion. Without WASM files, all tests will fail with module not found errors.
127+
128+
## Validation Scenarios
129+
130+
After making changes, ALWAYS run these validation scenarios:
131+
132+
### 1. Build Validation
133+
```bash
134+
npm run lint # Must pass before committing
135+
npm run build-ws # TypeScript compilation
136+
```
137+
138+
### 2. Package-Specific Validation
139+
For Graphviz changes:
140+
```bash
141+
cd packages/graphviz
142+
npm run build
143+
npm test
144+
# Test DOT rendering: echo 'digraph G { Hello -> World }' | npx graphviz-cli
145+
```
146+
147+
For DuckDB changes:
148+
```bash
149+
cd packages/duckdb
150+
npm run build
151+
npm test
152+
# Test SQL query functionality
153+
```
154+
155+
### 3. End-to-End Validation
156+
```bash
157+
# Test the examples in the root directory
158+
node -e "
159+
const { Graphviz } = require('./packages/graphviz/dist/index.js');
160+
Graphviz.load().then(g => console.log(g.version()));
161+
"
162+
```
163+
164+
## Known Limitations and Workarounds
165+
166+
### Build Environment Issues
167+
- **Network restrictions**: Downloads may fail in restricted environments
168+
- **Docker alternative**: Use `npm run build-docker` if local builds fail
169+
- **SSL certificate issues**: May block package downloads
170+
- **Node.js version**: Requires Node.js 20+ for optimal compatibility
171+
172+
### Package-Specific Notes
173+
174+
#### DuckDB Package
175+
- **Special build process**: Uses pre-built WASM from @duckdb/duckdb-wasm
176+
- **Can build without C++ compilation**: Run `npm run pack-duckdb` instead
177+
- **Build command**: Uses `pack-duckdb-eh` scripts to generate WASM files
178+
179+
#### Other Packages (base91, expat, graphviz, llama, zstd)
180+
- **Require C++ compilation**: Must run `npm run build-cpp` first
181+
- **WASM dependencies**: TypeScript builds fail without WASM files
182+
- **Build location**: WASM files generated in `build/packages/*/src-cpp/`
183+
184+
### Test Environment Setup
185+
```bash
186+
# Required for browser tests
187+
npx playwright install --with-deps
188+
189+
# Alternative for CI environments
190+
npx playwright install
191+
```
192+
193+
## File Structure Reference
194+
195+
```
196+
├── packages/ # Individual WASM packages
197+
│ ├── base91/
198+
│ │ ├── src/ # TypeScript source
199+
│ │ ├── src-cpp/ # C++ source
200+
│ │ ├── tests/ # Test files
201+
│ │ └── package.json
202+
│ ├── graphviz/ # Similar structure
203+
│ └── ...
204+
├── build/ # Generated WASM files
205+
│ └── packages/
206+
│ └── */src-cpp/*.wasm
207+
├── scripts/ # Build automation
208+
│ ├── cpp-build.sh
209+
│ ├── cpp-install-emsdk.sh
210+
│ └── cpp-install-vcpkg.sh
211+
├── .github/
212+
│ ├── workflows/ # CI pipelines
213+
│ └── instructions/ # Additional documentation
214+
├── emsdk/ # Emscripten SDK (generated)
215+
├── vcpkg/ # C++ package manager (generated)
216+
└── node_modules/ # npm dependencies
217+
```
218+
219+
## Common Commands Reference
220+
221+
### Maintenance
222+
```bash
223+
npm run clean-all # Clean all build artifacts
224+
npm run update # Update dependencies
225+
npm run lint-fix # Auto-fix linting issues
226+
```
227+
228+
### Documentation
229+
```bash
230+
npm run gen-docs # Generate TypeDoc documentation
231+
npm run build-docs # Build VitePress documentation
232+
```
233+
234+
### Advanced Build Options
235+
```bash
236+
npm run build-cpp-watch # Watch C++ files for changes
237+
npm run build-ts-watch # Watch TypeScript files
238+
npm run build-dev # Development builds
239+
```
240+
241+
## CI/Build Pipeline Notes
242+
243+
The `.github/workflows/test-pr.yml` pipeline runs:
244+
1. `npm ci` (3-5 minutes)
245+
2. `npm run install-build-deps` (20-30 minutes)
246+
3. `npm run lint` (2-3 minutes)
247+
4. `npm run build` (30-45 minutes)
248+
5. `npm run test` (15-20 minutes)
249+
6. `npm run test-bundlers` (5-10 minutes)
250+
251+
**Total CI time: 75-110 minutes** - This is NORMAL and expected.
252+
253+
## Debugging Build Issues
254+
255+
### WASM File Missing Errors
256+
```
257+
✘ [ERROR] Could not resolve "../../../build/packages/base91/src-cpp/base91lib.wasm"
258+
```
259+
**Solution**: Run `npm run build-cpp` to generate WASM files.
260+
261+
### Emscripten Not Found
262+
```
263+
CMake Error: Emscripten.cmake toolchain file not found
264+
Call Stack (most recent call first):
265+
buildtrees/0.vcpkg_dep_info.cmake:41 (vcpkg_triplet_file)
266+
```
267+
**Solution**:
268+
1. Run `npm run install-emsdk`
269+
2. Source environment: `source ./emsdk/emsdk_env.sh`
270+
3. Retry vcpkg install
271+
272+
### Emscripten Download Failures
273+
```
274+
Error: Downloading URL 'https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/node-v22.16.0-linux-x64.tar.xz': HTTP Error 403: Forbidden
275+
```
276+
**Solution**: Network restrictions may prevent downloads. Try:
277+
1. Use Docker build: `npm run build-docker`
278+
2. Or manually download and install Emscripten SDK
279+
280+
### Playwright Browser Missing
281+
```
282+
browserType.launch: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium_headless_shell-1181/chrome-linux/headless_shell
283+
```
284+
**Solution**: Run `npx playwright install --with-deps`
285+
286+
### Playwright Download Failures
287+
```
288+
Error: Download failed: size mismatch, file size: 180888452, expected size: 0
289+
```
290+
**Solution**: Network/proxy issues. Try:
291+
1. `PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm ci`
292+
2. Manually install browsers later
293+
3. Skip browser tests if not needed
294+
295+
### Module Not Found in Tests
296+
```
297+
Cannot find module '@hpcc-js/wasm-graphviz'
298+
✗ Base91 test failed: Cannot find module '@hpcc-js/wasm-base91'
299+
```
300+
**Solution**: Build packages first with `npm run build-ws`
301+
302+
### Bundle Test Failures
303+
```
304+
ERROR: "bundle" exited with 1.
305+
[!] (plugin commonjs--resolver) Error: Could not load /packages/base91/dist/index.js: ENOENT: no such file or directory
306+
```
307+
**Solution**: Complete full build cycle before running bundle tests.
308+
309+
## Development Workflow Summary
310+
311+
For new contributors:
312+
1. **Fresh clone**: `git clone && cd hpcc-js-wasm`
313+
2. **Dependencies**: `npm ci` (3 minutes)
314+
3. **Build tools**: `npm run install-build-deps` (30 minutes, NEVER CANCEL)
315+
4. **Full build**: `npm run build` (45 minutes, NEVER CANCEL)
316+
5. **Validation**: `npm run test` (20 minutes, NEVER CANCEL)
317+
318+
For routine development:
319+
1. **TypeScript changes**: `npm run build-ws && npm run lint`
320+
2. **C++ changes**: `npm run build-cpp && npm run build-ws` (45+ minutes)
321+
3. **Before commit**: `npm run lint` (required for CI)
322+
4. **Full validation**: `npm run test` (when ready for PR)
323+
324+
**Remember**: Set timeouts appropriately and NEVER CANCEL long-running build operations.

0 commit comments

Comments
 (0)