Problem using the result of yarn build:web

[epstein6]

Hello,

We are trying to add a new schema to DataHarmonizer and then build it for use in gitlab pages. We have successfully added our schema and it works when running yarn dev, but it errors when doing yarn build:web and then serving the resulting dist directory.

$ yarn build:web
yarn run v1.22.22
$ yarn clean:web && webpack --mode=production --config web/webpack.config.js && yarn build:schemas
$ rimraf web/dist
assets by status 206 MiB [big]
  assets by path templates/ 197 MiB
    assets by path templates/canada_covid19/ 861 KiB 2 assets
    + 10 assets
  assets by path images/locale/en/*.gif 6.05 MiB 12 assets
  asset scripts/main.js 3.02 MiB [emitted] [minimized] [big] (name: main) 1 related asset
assets by path templates/test/ 485 KiB
  asset templates/test/locales/fr/schema.json 174 KiB [emitted] [from: templates/test/locales/fr/schema.json] [copied]
  asset templates/test/schema.json 156 KiB [emitted] [from: templates/test/schema.json] [copied]
  asset templates/test/locales/de/schema.json 156 KiB [emitted] [from: templates/test/locales/de/schema.json] [copied]
assets by path *.html 1.12 KiB
  asset index.html 753 bytes [emitted]
  asset main.html 398 bytes [emitted] [from: main.html] [copied]
+ 10 assets
orphan modules 3.45 MiB [orphan] 799 modules
runtime modules 2.64 KiB 9 modules
built modules 7.47 MiB [built]
  modules by path ../node_modules/ 4.5 MiB 376 modules
  modules by mime type image/svg+xml 2.78 KiB 12 modules
  modules by path ../lib/ 31.7 KiB 7 modules
  modules by mime type image/png 1.08 KiB 4 modules
  modules by path ./ 2.94 MiB
    ./index.js + 408 modules 2.94 MiB [not cacheable] [built] [code generated]
    ../node_modules/css-loader/dist/cjs.js!./index.css 2.08 KiB [built] [code generated]
  modules by mime type image/gif 2.4 KiB
    data:image/gif;base64,R0lGODlhAQABAIAA.. 92.3 bytes [built] [code generated]
    data:image/gif;base64,R0lGODlhKAAoAIAB.. 2.31 KiB [built] [code generated]
  fs (ignored) 15 bytes [built] [code generated]
  stream (ignored) 15 bytes [built] [code generated]

WARNING in asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets: 
  scripts/main.js (3.02 MiB)
  images/locale/en/carousel_changeTemplate.gif (505 KiB)
  images/locale/en/carousel_exportingFiles.gif (397 KiB)
  images/locale/en/carousel_fillColumn.gif (382 KiB)
  images/locale/en/carousel_jumpToColumn.gif (266 KiB)
  images/locale/en/carousel_moreInfo.gif (397 KiB)
  images/locale/en/carousel_provenance.gif (339 KiB)
  images/locale/en/carousel_selectingVals.gif (311 KiB)
  images/locale/en/carousel_showRows.gif (424 KiB)
  images/locale/en/carousel_showSection.gif (282 KiB)
  images/locale/en/carousel_toggleRequiredCols.gif (1.13 MiB)
  images/locale/en/carousel_validatingCells.gif (631 KiB)
  images/locale/en/carousel_versionUpdate.gif (1.07 MiB)
  templates/MIxS/schema.json (187 MiB)
  templates/b2b2b/schema.json (629 KiB)
  templates/canada_covid19/schema.json (479 KiB)
  templates/grdi/schema.json (1.84 MiB)
  templates/grdi_1m/schema.json (624 KiB)
  templates/hpai/schema.json (1.87 MiB)
  templates/influenza/schema.json (279 KiB)
  templates/mpox/schema.json (1.21 MiB)
  templates/pha4ge/schema.json (474 KiB)
  templates/phac_dexa/schema.json (328 KiB)
  templates/wastewater/schema.json (1.48 MiB)
  templates/canada_covid19/locales/fr/schema.json (382 KiB)

WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  main (3.02 MiB)
      scripts/main.js


WARNING in webpack performance recommendations: 
You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
For more info visit https://webpack.js.org/guides/code-splitting/

webpack 5.93.0 compiled with 3 warnings in 23857 ms
$ yarn clean:schemas && webpack --config web/webpack.schemas.js
$ rimraf web/dist/dist-schemas web/dist/templates
assets by path *.js 130 MiB
  assets by status 129 MiB [big]
    asset 141.js 122 MiB [emitted] [minimized] [big]
    asset 938.js 1.34 MiB [emitted] [minimized] [big]
    asset 274.js 1.31 MiB [emitted] [minimized] [big]
    asset 115.js 1.08 MiB [emitted] [minimized] [big]
    + 6 assets
  + 34 assets
assets by path ../templates/ 8.51 MiB 43 assets
runtime modules 7.85 KiB 10 modules
orphan modules 23.2 KiB [orphan] 1 module
optional modules 130 MiB [optional]
  json modules 130 MiB
    modules by path ./templates/test/ 337 KiB 3 modules
    modules by path ./templates/canada_covid19/ 598 KiB 2 modules
    + 18 modules
  javascript modules 171 KiB
    ./templates/MIxS/export.js 128 bytes [optional] [built] [code generated]
    ./templates/SDRF/export.js 62 bytes [optional] [built] [code generated]
    + 18 modules
modules by path ./templates/ lazy ^/.//.*// 320 bytes
  ./templates/ lazy ^\.\/.*\/schema\.json$ namespace object 160 bytes [built] [code generated]
  ./templates/ lazy ^\.\/.*\/export\.js$ namespace object 160 bytes [built] [code generated]
./schemas.js + 1 modules 23.5 KiB [built] [code generated]

WARNING in configuration
The 'mode' option has not been set, webpack will fallback to 'production' for this value.
Set 'mode' option to 'development' or 'production' to enable defaults for each environment.
You can also set it to 'none' to disable any default behavior. Learn more: https://webpack.js.org/configuration/mode/

WARNING in asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets: 
  141.js (122 MiB)
  610.js (505 KiB)
  69.js (292 KiB)
  538.js (327 KiB)
  274.js (1.31 MiB)
  495.js (448 KiB)
  938.js (1.34 MiB)
  922.js (945 KiB)
  547.js (336 KiB)
  115.js (1.08 MiB)
  ../templates/canada_covid19/SOP.pdf (415 KiB)
  ../templates/grdi/SOP.pdf (647 KiB)
  ../templates/grdi_1m/SOP.pdf (597 KiB)
  ../templates/mpox/SOP_Mpox.pdf (592 KiB)
  ../templates/mpox/SOP_Mpox_international.pdf (581 KiB)
  ../templates/pha4ge/SOP.pdf (463 KiB)
  ../templates/b2b2b/schema.yaml (484 KiB)
  ../templates/hpai/schema.yaml (735 KiB)
  ../templates/grdi_1m/schema.yaml (313 KiB)
  ../templates/grdi/schema.yaml (778 KiB)
  ../templates/mpox/schema.yaml (805 KiB)
  ../templates/wastewater/schema.yaml (629 KiB)

webpack 5.93.0 compiled with 2 warnings in 13516 ms
✨  Done in 47.47s.

Uncaught ReferenceError: schemas is not defined
    <anonymous> http://[::1]:8000/scripts/main.js:2
    <anonymous> http://[::1]:8000/scripts/main.js:52
    <anonymous> http://[::1]:8000/scripts/main.js:52
main.js:2:2032977

Is there something we might be doing wrong here or anything else we need to edit when adding a new schema?

Thank you in advance!

[ddooley]

I don't have any immediate answer, expect perhaps that some file size limit is showing up in loading process - perhaps your MIxS/schema.json ? We haven't run into any schema file size limits yet but never tested with a 187Mb file!!!

Are you using master branch or the recommended dh2-1m-rework branch?
I'm able to do yarn build:web on my local dh2-1m-rework branch, but not seeing any errors. I have the following which is smaller than your templates/ folder. I'm a little puzzled by that since your listing doesn't show any other templates besides the stock DH /web/templates/ ones?

My local dh2-1m-rework

assets by status 22.6 MiB [big]
assets by path images/locale/en/*.gif 6.05 MiB 12 assets
assets by path templates/ 13.2 MiB

Yours has:
templates/MIxS/schema.json (187 MiB)

which is a giant extension of MIxS ???

Is a single schema.json packed in there?

[epstein6]

I believe we are using the main branch. I will try and rebase on dh2-1m-rework to see if it helps. It may be a problem with a size limit, as I am now noticing that we don't have a MIxS directory in the dist dir. I believe we are packing all the schemas from here (https://github.com/GenomicsStandardsConsortium/mixs/blob/main/src/mixs/schema/mixs.yaml) into a single json (using the linkml tools), is there a more recommended process for this, or is there already a version of DataHarmonizer that includes the MIxS schemas?

[ddooley]

There has been a MiXS DataHarmonizer compatible schema developed. You can see it most directly at: https://microbiomedata.github.io/submission-schema/_playground/ which takes in the NMDC LinkML spec, which contains their version of the GSC MIxS specification: https://microbiomedata.github.io/submission-schema/ . I think their version is a superset of MIxS 6.1 spec, but I can verify.

(The part of the schema that comes with the dh2-1m-rework menu is MiXS soil and a few other modules which originate from this.)

Its creator did this work a while back so I think the above uses a more official version of the MIxS spec.
https://turbomam.github.io/mixs-subset-examples-first/
https://github.com/turbomam/mixs-subset-examples-first
and Mark has a dataharmonizer setup to look at each of the MIxS subsets is:
https://github.com/turbomam/mixs-subset-examples-first/tree/main/data_harmonizer
The raw schema is: https://github.com/turbomam/mixs-subset-examples-first/blob/main/src/mixs_subset_examples_first/schema/mixs_subset_examples_first.yaml

If you have other q's I can probably get them answered...

Cheers,
Damion

[epstein6]

OK, I have been starting over from square one (removing our modifications) for the yarn build:web. I checked out out the dh2-1m-rework branch, and ran yarn install and yarn build:web, but when I load it in the browser I get

Error: Unable to load file: "/templates/canada_covid19/schema.json" - received error: 404

Is there something I may be doing wrong here? The build process doesn't seem to create a schema.json in the dist directory.

[ddooley]

The build process converts all of the referenced schema.json files into compressed javascript import files into /web/dist/dist-schemas/ under different names. The only verbatim file that gets copied into templates is the schema.yaml one.

There is the "yarn" step before "yarn:build" IF switching between master and dh2-1m-rework branches...

Were you modifying the menu.json file to suit your needs? Basically if a schema & template combo is mentioned in menu.json (with display:true) then its schema.json file must be in /web/templates/[schema folder]/schema.json

I.e. have you done any customizations to template folders? One can start with an empty menu.json, and build up only the templates that should show in the generated DataHarmonizer by running /scripts/tabular_to_schema.py -m in a given template folder - this adds that template to the menu.json file correctly.

LMK!

[ddooley]

Also, last year we created a "yarn build:schemas" step as an option to compile the schemas separately from data harmonizer. This generates the /web/dist/dist-schemas/schemas.js file. One still has to do the "yarn build:web" step to generate the index.html etc. interface (which also does "build:schemas" inside it). This uncouples building the html interface from the menu of schemas. It still relies on menu.js to direct what schemas to include or exclude from the /web/templates/ folders though.

[epstein6]

For this latest test I just checked out the dh2-1m-rework branch (no customizations), ran yarn, ran yarn build:web, and then served the files with python -m http.server in the web/dist directory. This results in the error about loading the "schema.json" file. Is there something I am doing wrong here? Is that the proper procedure for building static files, and is there something wrong with the file generation?

[ddooley]

Interesting! I actually haven't recently tested DataHarmonizer via a webserver. The software was designed to work offline, i.e. just browse in file menu (e.g. mac finder) to the /web/dist/ folder and click on index.html and it should all work.
I'll try to see what is going on via python http.server later today.

[ddooley]

Ok, I just tested and saw the error you encountered, oy, that is a bug! We are so used to having DH used in an off-line environment, that I can't guess when this might have started happening. In meantime you can test DH without a webserver, ok?

[epstein6]

Glad that it is reproducible! We would really like to be able to use this as a hosted thing on our network so that users don't need to download it, but we can at least try this for now.

Thanks!

[ddooley]

Its definitely meant to be hostable!

[ddooley]

This should be fixed now on the dh2-1m-rework branch. Templates.js fetchSchema() call logic was modified.
LMK if that works...

[epstein6]

Unfortunately when I upload the dist files to our file hosting it still gives me the error. It seems to work when I run the python test webserver on localhost though.

[ddooley]

I'm having a teammate test out on a webserver in the next few days. In the meantime one other experiment would be to simplify this templates.js function:

async function fetchSchema(path) {

  if (window.location.href.startsWith('http://localhost:') || window.location.protocol.startsWith('file')) {
    const schema_path = path.replace(/\/templates\/(.+)\/schema.json/, '$1');
    return await getSchema(schema_path);
  } 
  else if (window.location.protocol.startsWith('http')) {

    // PROBLEM: local HTTPS server in /dist folder doesn't work
    // path e.g. /templates/canada_covid19/schema.json
    return await fetchFileAsync(path);
  }

  return null;
}

to

async function fetchSchema(path) {

    const schema_path = path.replace(/\/templates\/(.+)\/schema.json/, '$1');
    return await getSchema(schema_path);

}

But this will disable the capacity to load a template schema from the file menu so we'd have to fix that menu option!

[epstein6]

Great, this fixes the error in my setup! We don't use the load template schema menu option so that'll be OK for us.