Skip to content

Commit b22e85b

Browse files
authored
Merge pull request #126 from budhash/feature/bengali-web-npm
Expose Bengali through npm WASM and browser language selection
2 parents 22a3696 + 4500b80 commit b22e85b

17 files changed

Lines changed: 381 additions & 64 deletions

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,3 +50,6 @@ mined_candidates_hi.csv
5050

5151
# Parser regression baseline (ephemeral; regenerate with make regression-baseline)
5252
.regression-baseline.tsv
53+
54+
# Copied with the WASM demo assets from npm/NOTICE.md
55+
/web/NOTICE.md

‎.goreleaser.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,9 @@ archives:
3939
files:
4040
- README.md
4141
- LICENSE
42+
# Binaries embed CC BY-SA / CC BY data; ship the attribution notice.
43+
- src: npm/NOTICE.md
44+
dst: NOTICE.md
4245

4346
checksum:
4447
name_template: 'checksums.txt'

‎CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,17 @@ tagged release also has auto-generated notes on the
1313

1414
### Changed
1515

16+
- npm/WASM accepts per-call `language: "bengali"`; omitted language remains Hindi.
17+
The browser adds explicit language selection and remembers options per language.
18+
Unsupported JavaScript language values throw without terminating the WASM engine.
19+
1620
- CLI accepts `--language=bengali` or `--language bengali` for every input mode;
1721
omitted language remains Hindi. Bengali learned components stay opt-in.
1822
Unsupported names fail with the supported list; Bengali `--rerank` without
1923
`--schwa-model` prints a warning instead of being silently ignored.
24+
- npm wrappers reject non-object options and non-boolean flags (a BigInt flag
25+
previously crashed the WASM runtime). The npm license field and release
26+
archives now carry the embedded-data licenses (`NOTICE.md`).
2027

2128
- Debug rule traces now retain metadata-only changes in all phases and fallback
2229
passes; metadata extraction remains disabled outside debug execution.

‎Makefile‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ wasm: ## Build the WebAssembly demo into web/ (gomanize.wasm + wasm_exec.js + ve
7272
@cp "$$($(GOCMD) env GOROOT)/lib/wasm/wasm_exec.js" $(WASM_DIR)/wasm_exec.js 2>/dev/null \
7373
|| cp "$$($(GOCMD) env GOROOT)/misc/wasm/wasm_exec.js" $(WASM_DIR)/wasm_exec.js
7474
@cp $(NPM_DIR)/index.mjs $(WASM_DIR)/gomanize.mjs
75+
@cp $(NPM_DIR)/NOTICE.md $(WASM_DIR)/NOTICE.md
7576
@echo "✓ WASM demo built. Serve locally with: make wasm-serve"
7677

7778
wasm-serve: wasm ## Build and serve the WASM demo at http://localhost:8080
@@ -87,9 +88,9 @@ npm: wasm ## Assemble the @budhash/gomanize npm package (copies wasm + wasm_exec
8788
@cp $(WASM_DIR)/wasm_exec.js $(NPM_DIR)/dist/wasm_exec.js
8889
@echo "✓ npm package ready in $(NPM_DIR)/"
8990

90-
npm-test: npm ## Build + smoke-test the npm package under Node
91+
npm-test: build npm ## Build + smoke-test the npm package under Node
9192
@echo "Smoke-testing npm package..."
92-
@node $(NPM_DIR)/smoke.test.mjs
93+
@GOMANIZE_CLI="$(CURDIR)/$(BINARY)" node $(NPM_DIR)/smoke.test.mjs
9394

9495
install: build ## Install gomanize to GOPATH/bin
9596
@echo "Installing $(BINARY)..."

‎README.md‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -168,8 +168,8 @@ The Go library accepts `gomanize.New("bengali")`. Experimental B1 adds scoped
168168
phalas, positional conjuncts, final-cluster vowels, and হও/হওয়া handling to the
169169
B0 symbols and Unicode aliases. It reaches 56.52% match-any on the held-out
170170
Dakshina word set; pronunciation ambiguities remain. The CLI accepts `--language=bengali` (or
171-
`--language bengali`); omitting it preserves Hindi. npm and web-demo language
172-
selection will follow in a separate change. See the [B1 results and
171+
`--language bengali`); omitting it preserves Hindi. npm accepts `{ language: "bengali" }`, and the browser has an explicit language
172+
selector. Both continue to default to Hindi. See the [B1 results and
173173
limitations](docs/reviews/2026-10-04-bengali-b1-results.md).
174174

175175

@@ -203,4 +203,12 @@ gomanize --language=bengali --list-rules
203203
Language selection also applies to `--test`, `--debug`, and rule overrides.
204204
Only `hindi` and `bengali` are accepted (case-insensitive); unsupported or empty
205205
languages fail with a nonzero exit. Bengali reranking needs `--schwa-model`;
206-
style flags retain the library's documented learned-component fallbacks.
206+
Hindi style flags do not implement Bengali rendering styles; supplying them
207+
bypasses Bengali lexicon lookup and reranking. The browser hides those Hindi
208+
style controls while Bengali is selected.
209+
210+
```js
211+
const g = await load();
212+
g.translit("আমি বাংলা", { language: "bengali" }); // ami bangla
213+
g.translit("नमस्ते दुनिया"); // namaste duniya — default remains Hindi
214+
```

‎TASKS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,8 +60,8 @@ Tags: `@deps=` `@rel=` `@branch=` `@pr=` `@issue=` `@tags=` `@effort=` `@system=
6060
- [x] (T-0066) [P2] [done] B2 promote cross-fitted selector: native-aware runtime contract, parity, style fallbacks and external validation @deps=T-0065 @branch=feature/bengali-b2-runtime-rerank @rel=T-0054 @done=2026-10-05 @pr=123
6161
- [x] (T-0067) [P2] [done] B3 source-pinned Bengali lyrics pilot, provisional references and line-level evaluation @deps=T-0066 @branch=feature/bengali-b3-lyrics-seed @done=2026-10-05 @pr=124
6262
- [ ] (T-0068) [P2] [todo] B3 independent Bengali reference review: correct and attest the lyrics pilot before gold promotion @deps=T-0067
63-
- [x] (T-0069) [P2] [done] Expose Bengali in CLI with explicit language selection and Hindi-default compatibility @deps=T-0067 @branch=feature/bengali-cli @done=2026-10-05
64-
- [ ] (T-0070) [P2] [todo] Expose Bengali through npm/WASM and browser language selection with Hindi defaults @deps=T-0069
63+
- [x] (T-0069) [P2] [done] Expose Bengali in CLI with explicit language selection and Hindi-default compatibility @deps=T-0067 @branch=feature/bengali-cli @done=2026-10-05 @pr=125
64+
- [x] (T-0070) [P2] [done] Expose Bengali through npm/WASM and browser language selection with Hindi defaults @deps=T-0069 @branch=feature/bengali-web-npm @done=2026-10-05 @pr=126
6565
- [ ] (F-0009) [P2] [todo] npm distribution: @budhash/gomanize (WASM engine + JS/TS wrapper) @shadow
6666
- [x] (T-0036) [P2] [done] Package the WASM engine as @budhash/gomanize with loader, types, make target, and Node smoke test @done=2026-09-08
6767
- [ ] (F-0006) [P2] [todo] WASM build + web demo (data flywheel) @shadow

‎cmd/gomanize-wasm/main.go‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,9 @@
55
//
66
// gomanizeTranslit(text, options) -> string
77
//
8-
// where options is an object of booleans (see webdemo.FlagNames). The engine
8+
// where options contains boolean flags (see webdemo.FlagNames) and an optional
9+
// language: "hindi" (default) or "bengali". Invalid languages return a JS Error
10+
// which the npm wrapper throws. The engine
911
// and all its embedded models are compiled in via go:embed, so the page needs
1012
// no network calls beyond fetching the .wasm itself.
1113
//
@@ -14,16 +16,21 @@
1416
package main
1517

1618
import (
19+
"strings"
1720
"syscall/js"
1821

1922
"github.com/budhash/gomanize"
2023
"github.com/budhash/gomanize/webdemo"
2124
)
2225

2326
func main() {
24-
g, err := gomanize.New("hindi")
25-
if err != nil {
26-
panic(err)
27+
engines := make(map[string]*gomanize.Gomanize)
28+
for _, language := range []string{"hindi", "bengali"} {
29+
g, err := gomanize.New(language)
30+
if err != nil {
31+
panic(err)
32+
}
33+
engines[language] = g
2734
}
2835

2936
// The WASM runtime dispatches JS calls one at a time on a single event
@@ -34,13 +41,25 @@ func main() {
3441
}
3542
text := args[0].String()
3643

44+
language := "hindi"
3745
flags := make(map[string]bool, len(webdemo.FlagNames()))
3846
if len(args) >= 2 && args[1].Type() == js.TypeObject {
47+
value := args[1].Get("language")
48+
if value.Type() != js.TypeUndefined {
49+
if value.Type() != js.TypeString {
50+
return js.Global().Get("Error").New("gomanize: language must be hindi or bengali")
51+
}
52+
language = strings.ToLower(value.String())
53+
}
3954
for _, name := range webdemo.FlagNames() {
4055
flags[name] = args[1].Get(name).Truthy()
4156
}
4257
}
4358

59+
g, ok := engines[language]
60+
if !ok {
61+
return js.Global().Get("Error").New("gomanize: unsupported language: " + language)
62+
}
4463
g.SetOptions(webdemo.Options(flags))
4564
return g.Translit(text)
4665
})
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Bengali CLI, npm/WASM and browser integration
2+
3+
T-0069 exposes the existing Bengali engine through CLI language selection;
4+
T-0070 adds the same choice to npm/WASM and the browser. Neither increment
5+
changes the core rules, learned artifacts or quality thresholds. Bengali
6+
remains experimental; the independent lyrics-reference review is still open.
7+
8+
## Contract
9+
10+
- CLI: `--language=bengali` and `--language bengali`; omitting the option selects
11+
Hindi. Selection applies to every input/debug/test/rule mode. Full language
12+
names are case-insensitive. Missing, empty and unsupported values fail.
13+
- JS: `translit(text, {language: "bengali"})`; each call independently defaults
14+
to Hindi when language is absent. Both ESM and CJS forward the option. Invalid
15+
values throw an Error without terminating the WASM runtime. The TypeScript
16+
`Language` union exposes the canonical lowercase names.
17+
- Browser: a labeled selector updates input language, label, placeholder, sample
18+
and rendered output. Existing text survives language changes. Checkbox values
19+
are remembered separately per language for the current page session. Hindi
20+
retains its previous Long vowels default; Bengali starts with all flags off.
21+
- Bengali reranking requires the vowel model. Hindi style toggles are hidden
22+
for Bengali: they do not implement Bengali rendering styles, and passing them
23+
through the API bypasses Bengali lexicon/reranking under the existing contract.
24+
- npm and web assets include code/embedded-data attribution via `NOTICE.md`.
25+
The package version is unchanged and no package or website is published.
26+
27+
## Validation
28+
29+
CLI subprocess tests run the actual entry point and compare both languages with
30+
the library. Coverage includes positional and piped input, input files, TSV
31+
comparison, debug, rule listing/overrides, omitted language and malformed flags.
32+
A sanity-revert to the hard-coded Hindi constructor fails the Bengali tests.
33+
Full local `make ci` passed for CLI commit `3e3665e` in an isolated checkout.
34+
35+
`make npm-test` now builds the CLI as well as WASM. Both ESM and CJS test nine
36+
option profiles for each language against the CLI, plus defaults after Bengali
37+
calls, option reset, mixed scripts, whitespace/punctuation and recovery after
38+
invalid languages. A sanity-revert that ignores the WASM language option fails
39+
its Bengali assertion; the restored implementation passes. TypeScript strict
40+
checking accepts both languages and rejects an unsupported one. An npm pack
41+
dry run confirms the entry points, declarations, WASM, runtime and notice ship.
42+
43+
Local Safari inspection confirms initial Hindi selection, successful Bengali
44+
sample rendering, text preservation during switching, restored Hindi defaults
45+
and remembered Bengali learned options. A final page check confirms that only
46+
supported learned controls appear for Bengali. The second full local `make ci`
47+
result is recorded in the npm/browser PR. Hosted full CI remains configured for
48+
main-targeting PRs, so these stacked-branch full runs are local.
49+
50+
## Review notes (2026-10-05)
51+
52+
Independent stack review; language/option isolation, ESM/CJS parity, textContent
53+
output and per-language UI state verified.
54+
55+
- **Flag types are validated in JS.** Only `language` was type-checked, so a
56+
BigInt flag (`{longVowels: 1n}`) panicked inside the Go runtime and every later
57+
call failed with "Go program has already exited"; string flags were truthy
58+
(`"false"` meant true). Both wrappers now reject non-object options and
59+
non-boolean flags with `TypeError`; the smoke test asserts each case and that
60+
the instance survives.
61+
- **Licensing.** `NOTICE.md` lists every embedded data file (Hindi lexicon, schwa
62+
tree, n-grams; Bengali lexicon, vowel tree, selector) with source, license and
63+
changes. `package.json` declares `MIT AND CC-BY-SA-4.0 AND CC-BY-4.0`. GoReleaser
64+
archives now include `NOTICE.md`; the npm README links it absolutely.
65+
- **Web.** The language select opts out of browser form restoration so a restored
66+
value cannot desynchronize the UI; structured data lists `bn`.
67+
- Tracked: unknown option keys are ignored; npm Bengali `rerank` without
68+
`schwaModel` is silent (the CLI warns); the About dialog still describes the
69+
Hindi components only.

‎npm/NOTICE.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Code and embedded-data attribution
2+
3+
Gomanize implementation code is MIT licensed, Copyright (c) 2023–2026
4+
Budhaditya (budhash@gmail.com). The WebAssembly binary also embeds learned data;
5+
the code license does not replace the data licenses below.
6+
7+
Every gomanize build (CLI binaries, the WebAssembly module, the npm package and
8+
the web demo) embeds the data files below:
9+
10+
| Embedded file | Derived from | License | Changes from the source |
11+
|---|---|---|---|
12+
| `lang/hindi/lexicon.tsv` | Dakshina Hindi train spellings; Shabd frequencies (CC0) for ranking | CC BY-SA 4.0 | High-attestation winning spelling per word, frequency-filtered, serialized as TSV |
13+
| `lang/hindi/schwa_tree.json` | Dakshina Hindi train alignments | CC BY-SA 4.0 | Decision tree learned over aligned schwa positions |
14+
| `lang/hindi/roman_ngrams.tsv` | Dakshina Hindi train romanizations | CC BY-SA 4.0 | Character 4-gram counts |
15+
| `lang/bengali/lexicon.tsv` | Dakshina Bengali train spellings | CC BY-SA 4.0 | Normalized native keys, held-out vocabularies excluded, unique high-attestation winners, TSV |
16+
| `lang/bengali/vowel_tree.json` | Google Bengali pronunciation lexicon (train partition) | CC BY 4.0 | Grapheme–phoneme alignment; inherent-vowel decision tree |
17+
| `lang/bengali/selector.json` | Dakshina Bengali train spellings and the Google Bengali lexicon | CC BY-SA 4.0 | Out-of-fold candidate generation; preference tree |
18+
19+
Sources:
20+
21+
- **Google Dakshina**, Google / Roark et al., 2020:
22+
[source](https://github.com/google-research-datasets/dakshina),
23+
[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
24+
- **Google Bengali pronunciation lexicon**, Copyright Google 2015/2016:
25+
[source](https://github.com/googlei18n/language-resources/tree/master/bn),
26+
[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
27+
- **Shabd** psycholinguistic database: [source](https://osf.io/xfbhd/),
28+
[CC0](https://creativecommons.org/publicdomain/zero/1.0/) (no attribution
29+
required; credited for transparency).
30+
31+
Source hashes, transformations, exclusions and reproduction tools are recorded
32+
in the [training data record](https://github.com/budhash/gomanize/blob/main/training/data/bengali/README.md),
33+
[Bengali lexicon record](https://github.com/budhash/gomanize/blob/main/lang/bengali/LEXICON.md),
34+
[selector record](https://github.com/budhash/gomanize/blob/main/lang/bengali/RERANKER.md),
35+
and [research record](https://github.com/budhash/gomanize/blob/main/docs/RESEARCH.md).
36+
These notices identify adapted data and do not imply endorsement by its creators.
37+
38+
## MIT code license
39+
40+
MIT License
41+
42+
Copyright (c) 2023-2026 Budhaditya (budhash@gmail.com)
43+
44+
Permission is hereby granted, free of charge, to any person obtaining a copy
45+
of this software and associated documentation files (the "Software"), to deal
46+
in the Software without restriction, including without limitation the rights
47+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
48+
copies of the Software, and to permit persons to whom the Software is
49+
furnished to do so, subject to the following conditions:
50+
51+
The above copyright notice and this permission notice shall be included in all
52+
copies or substantial portions of the Software.
53+
54+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
55+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
56+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
57+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
58+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
59+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
60+
SOFTWARE.

‎npm/README.md‎

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# @budhash/gomanize
22

3-
Romanize Hindi (Devanagari) into readable, diacritic-free Latin script — the
3+
Romanize Hindi (Devanagari) or experimental Bengali into readable, diacritic-free Latin script — the
44
[gomanize](https://github.com/budhash/gomanize) engine (Go) compiled to
55
WebAssembly, with a thin JS/TS wrapper. It is the **same engine as the CLI**, so
66
output is identical; nothing is reimplemented.
@@ -20,28 +20,38 @@ import { load } from "@budhash/gomanize";
2020

2121
const g = await load(); // instantiate once
2222
g.translit("नमस्ते दुनिया"); // "namaste duniya"
23+
g.translit("আমি বাংলা", { language: "bengali" }); // "ami bangla"
2324
g.translit("गाना", { longVowels: true }); // "gaanaa"
24-
g.translit("मेरे सपनों की रानी", { rerank: true }); // "mere sapnon ki rani"
25+
g.translit("मेरे सपनों की रानी", { rerank: true }); // "mere sapanon ki rani"
2526
```
2627

2728
CommonJS works too: `const { load } = require("@budhash/gomanize");`
2829

2930
## Options
3031

31-
All optional, default `false`:
32+
Language defaults to `"hindi"` on every call; boolean flags default to `false`.
33+
Unsupported languages (including empty strings) throw `Error`; language names
34+
are case-insensitive at runtime. The TypeScript type uses canonical lowercase names.
3235

3336
| Group | Option | Effect |
3437
|-------|--------|--------|
35-
| rule tweak | `longVowels` | write "aa" for every ā (गाना → gaanaa) |
36-
| rule tweak | `simpleNasals` | करें → karen, not karein |
37-
| rule tweak | `keepMedialSchwa` | disable CCV deletion (जनता → janata) |
38+
| language | `language` | `"hindi"` (default) or `"bengali"` |
39+
| Hindi rule tweak | `longVowels` | write "aa" for every ā (गाना → gaanaa) |
40+
| Hindi rule tweak | `simpleNasals` | करें → karen, not karein |
41+
| Hindi rule tweak | `keepMedialSchwa` | disable CCV deletion (जनता → janata) |
3842
| learned | `schwaModel` | decision-tree schwa classifier |
3943
| learned | `lexicon` | prefer attested spellings |
40-
| learned | `rerank` | re-rank candidates with a char n-gram model |
44+
| learned | `rerank` | rank candidates; Bengali requires `schwaModel: true` |
45+
46+
For Bengali, `schwaModel` enables the three-way vowel model; `rerank` requires
47+
that option too. `lexicon` remains opt-in and can change spelling style. The
48+
three Hindi rule tweaks do not implement Bengali rendering styles; supplying
49+
any of them bypasses Bengali lexicon lookup and reranking. Language selection
50+
is explicit, not script detection. Input in other scripts is preserved.
4151

4252
## Notes
4353

4454
- `load()` is async (instantiates the wasm) and cached; `translit` is synchronous after.
45-
- The `.wasm` is ~1.4 MB gzipped — it embeds the schwa model, lexicon, and n-gram data.
55+
- The `.wasm` embeds both languages and their learned data; Bengali needs no additional download.
4656
- Runs on **Node ≥ 18** and modern browsers/bundlers (the wasm + `wasm_exec.js` are resolved relative to the module).
47-
- Romanization, not strict transliteration — see the [project README](https://github.com/budhash/gomanize). MIT licensed.
57+
- Romanization, not strict transliteration — see the [project README](https://github.com/budhash/gomanize). MIT code; embedded data has separate licenses in [NOTICE.md](https://github.com/budhash/gomanize/blob/main/npm/NOTICE.md) (shipped in this package).

0 commit comments

Comments
 (0)