feat(d2e-compat): run the d2e stack on trex's canonical main#92
Merged
Conversation
Port d2e's initTrex() blocks 1-7 into d2e-compat/boot.ts (exported as d2eBoot()). Copies lib/attach.ts verbatim (import-path fix only). Blocks 6-7 replace DatabaseManager with a direct trexdb.pool query + decryptSecret; portal.dataset cache_id enumeration is skipped with a logged warning (table absent from trex schema). All blocks are independently try/caught.
- /logto: EdgeRuntime userWorker spawn (degrades to 503 without EdgeRuntime)
- /oauth/token: Logto PKCE token exchange forwarded to LOGTO__TOKEN_URL
- /portal/plugin.json: public; degrades to {} (no global.PLUGINS_JSON in trex)
- /portal/env.js: public; builds client env from Deno.env at request time
- /trex/db/*: CRUD against trexdb.database + database_credential; requireAdmin
- /trex/log: audit POST, console-logged; requireAdmin
- requireAdmin: minimal admin gate from authz.ts (alp_role_system_admin flag
or client-credentials token); full scope parity deferred to parity phase
… in /trex/log; note shared JWKS
Restores the original d2e precedence for audit identity: 1. oid from nested thirdPartyToken (Azure AD) in the Logto JWT 2. GATEWAY__IDP_SUBJECT_PROP claim or "oid" from the Logto JWT 3. logtoSubject (verified sub) as final fallback requireAdmin now stashes the full verified payload as req.logtoPayload so the /trex/log handler can derive identity without re-verifying. The nested thirdPartyToken is decoded without signature verification, matching d2e's jwt.decode usage (the outer token is already verified).
…ues in token exchange - applyD2eCompat is now async/awaited so routes register before server.listen - Replace fire-and-forget import().then() with await import(); correct misleading sync comment - Update call site in core/server/index.ts to await applyD2eCompat(app) - Token exchange no longer logs response header values; logs names only to prevent credential leaks via Set-Cookie or aliased auth headers
…lt load path; guard d2e boot
V1 hardcoded GRANT ALL PRIVILEGES ON DATABASE testdb, which fails on any deployment whose DB isn't named testdb (e.g. d2e's 'alp'), aborting the migration before the control-plane tables (kek_wrapped_dek, ...) are usable and hanging the main at initDek.
scanPluginDirectory was given the whole PATH-style value and readDir'd it as a single directory, so a multi-entry path (d2e uses /usr/src/plugins-dev:/usr/src/bundled-plugins:/usr/src/plugins) silently scanned nothing -> 0 plugins active. Split on ':' and scan each entry.
trex scopes plugin routes under PLUGINS_BASE_PATH/<scope>/ (e.g. /plugins/trex/web). The d2e fork served its plugins at bare paths (/portal, /atlas, /analytics-svc, /system-portal, ...), and the d2e front-end + routing still expect those. Mount @trex plugins scoped as before, but mount everything else (d2e/@data2evidence/@OHDSI) at the bare source path so d2e UIs and their API calls resolve. Applies to both ui.ts (static) and function.ts (edge fns).
The d2e front-end/caddy prefix every request with /d2e (the d2e Hono fork stripped it in its Hono getPath). trex's Express main didn't, so /d2e/portal, /d2e/WebAPI, etc. hit Express unmatched -> 'Cannot GET /d2e/portal'. Add an early middleware (first, before any route) that rewrites /d2e/* -> /* on both url and originalUrl. Gated by D2E_COMPAT; base path overridable via D2E_BASE_PATH (default /d2e).
…loop) Stripping /d2e for routing also made trailing-slash redirects (express.static /portal -> /portal/) drop the prefix, so the client bounced out of the /d2e base path and ping-ponged with the caddy front door (ERR_TOO_MANY_REDIRECTS). Wrap res.setHeader on prefixed requests to re-add /d2e to root-relative Location targets, so /d2e/portal -> 301 -> /d2e/portal/ -> 200 (one hop).
d2e UI plugins (portal, ...) are React SPAs but their manifests use the
source/target dialect without spa:true, so trex registered them as plain static
and client routes 404'd ('Cannot GET /portal/login-callback' after OIDC login).
Treat root-mounted (non-@trex) routes as SPA: serve index.html for extension-less
sub-paths when the route ships an index.html (statSync-guarded for pure asset
dirs). Mirrors the d2e fork's /portal/* fallback.
The d2e portal loads /portal/env.js (runtime window.ENV_DATA) and
/portal/plugin.json (ui-plugins); a missing env.js crashes the app
('Cannot read properties of undefined (reading endsWith)') -> blank page.
Two causes: (1) the d2e-ui plugin's /portal static+SPA fallback shadowed those
dynamic routes because applyD2eCompat ran AFTER initPlugins -> register
D2E_COMPAT routes BEFORE plugins so the specific /portal/env.js & plugin.json
win; (2) plugin.json returned a {} stub -> serve ui.ts getPluginsJson(). Also
make the SPA fallback next() on asset paths instead of 404 so dynamic siblings
still resolve.
The Logto PKCE token exchange failed ('no client authentication mechanism
provided'): the form body never parsed (no urlencoded middleware; the manual
raw-stream read yielded nothing in the trexas runtime), so grant_type/code/
client_id never reached Logto. Add express.urlencoded() to the route. Send the
client_secret via the body only (client_secret_post) — also sending a Basic
header tripped Logto's 'client authentication must only be provided using one
mechanism'. Accept SECURITY_AUTH_OIDC_APISECRET or LOGTO__CLIENT_SECRET.
trex gated every function route with authContext+pluginAuthz; for d2e (@data2evidence) functions that 401'd Logto-authenticated calls (incl. the portal's public APIs) before they reached the worker. @trex plugins keep trex auth; d2e plugins authenticate inside the worker via the forwarded Logto Authorization header, as the d2e fork did.
d2e function workers build their `services` object (portalServer, webapi, ...) from SERVICE_ROUTES; the d2e fork injected it into every worker's env. trex only passed TREX_FUNCTION_PATH, so d2e workers threw 'No url is set for PortalAPI'. Inject SERVICE_ROUTES when set (no-op for @trex-only deployments).
… fixes - dbm-sync: make /trex/db a facade over Trex.DatabaseManager; push the trexdb registry into setCredentials() on every write and at boot so registered DBs attach as <id>__srcdb and analytics-svc resolves them - function.ts: decoratorType + host fs access so NestJS/typeorm function workers init; key fnmap by the unscoped plugin name so Trex.tokioChannel worker-to-worker calls resolve; provide DATABASE_CREDENTIALS to d2e workers - auth.ts: requireAdmin accepts tokens carrying roles:["role.systemadmin"] - index.ts: redirect / to /d2e/portal when D2E_COMPAT is on
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if (chunks.length) body = await new Blob(chunks as BlobPart[]).arrayBuffer(); | ||
| } | ||
| try { | ||
| const r = await fetch(target, { method: (req as any).method, headers, body: body as BodyInit | undefined, redirect: "manual" }); |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #92 +/- ##
========================================
Coverage 52.34% 52.34%
========================================
Files 167 167
Lines 71733 71733
========================================
Hits 37546 37546
Misses 34187 34187
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
- Return generic JSON error bodies instead of String(e) in /trex/db and worker-proxy handlers (stops stack-trace exposure and exception-text-as-HTML); full error is still logged server-side. - Harden the /WebAPI proxy: derive pathname+search from a fixed base and require a /WebAPI/ prefix so a crafted originalUrl can't redirect the proxy (SSRF).
DuckDB publishes the SQLite reader only as `sqlite_scanner`; there is no standalone `sqlite` extension (its download 404s). It was silently skipped before, but the fail-loud fetch added in fb2fd10 turned that 404 into a fatal build error. Remove `sqlite` from DUCKDB_CORE_EXTENSIONS; `sqlite_scanner` (which provides the alias) stays. trex never `LOAD`s a bare `sqlite`.
p-hoffmann
added a commit
to OHDSI/Data2Evidence
that referenced
this pull request
Jun 22, 2026
…ndored core PR OHDSI/trex#92 merged the env-gated D2E_COMPAT layer into trex's canonical main+event, so the trexsql image now ships it. Point Dockerfile.v2 at that image (sha-464e0fc) and remove the vendored services/trex/core overlay + bundle steps — the core (index.eszip) comes from the base image, with D2E_COMPAT activated via the D2E_COMPAT=true env in compose. Removes the now-redundant vendored core.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an env-gated
D2E_COMPATlayer inside trex's canonical Express main so the d2e stack (portal, WebAPI, Atlas3, d2e functions) runs on it — replacing d2e's forked Hono main. No behavior change whenD2E_COMPATis off.What's in it
d2e-compat/boot.ts)./WebAPIproxy + Logto authn,/logto,/oauth/token,/portal/{env.js,plugin.json},/trex/{db,log}.@trexplugin routes, colon-separatedPLUGINS_DEV_PATH, SPA fallback for d2e UIs,/d2ebase-prefix strip with redirect re-prefixing.decoratorType/host-fs access for NestJS/typeorm;fnmapkeyed by the unscoped plugin name soTrex.tokioChannelworker-to-worker calls resolve;SERVICE_ROUTES+DATABASE_CREDENTIALSprovided to d2e workers./trex/dbis now a facade over the trex-nativeTrex.DatabaseManager(dbm-sync.ts) — registered DBs are pushed tosetCredentials()on write + at boot and attached as<id>__srcdb.requireAdminacceptsroles:["role.systemadmin"]; root/→/d2e/portalunderD2E_COMPAT.Validation
Built FROM the trexsql image and run locally end-to-end: OIDC login, portal researcher dashboard, all
system-portal/*200, Atlas3 served, WebAPI up, DB registration attaches the source and analytics-svc resolves the credential.