Adopt jupyter-builder

[Darshan808]

Description

Update the extension template to use the new jupyter-builder package. This lets us perform build without needing whole jupyterlab.

This change is part of the ongoing work on https://github.com/jupyterlab/jupyter-builder under Jupyter CFP.

[Darshan808]

CI fails as expected due to missing @jupyter/builder package. Adding it in next commit.

[krassowski]

CI fails as expected due to missing @jupyter/builder package. Adding it in next commit.

Is it really expected? Or should jupyter-builder ship with @jupyter/builder assets already? I would have expected them to be used here, but maybe I am wrong.

      INFO:hatch_jupyter_builder.utils:> /tmp/pip-build-env-xhu8ml_k/overlay/bin/jlpm run build:prod
      Building extension in .
      node:internal/modules/cjs/loader:1459
        throw err;
        ^
      
      Error: Cannot find module '/home/runner/work/extension-template/extension-template/myextension/node_modules/@jupyter/builder/lib/build-labextension.js'
          at Module._resolveFilename (node:internal/modules/cjs/loader:1456:15)
          at defaultResolveImpl (node:internal/modules/cjs/loader:1066:19)
          at resolveForCJSWithHooks (node:internal/modules/cjs/loader:1071:22)
          at Module._load (node:internal/modules/cjs/loader:1242:25)
          at wrapModuleLoad (node:internal/modules/cjs/loader:255:19)
          at Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:154:5)
          at node:internal/main/run_main_module:33:47 {
        code: 'MODULE_NOT_FOUND',
        requireStack: []

Irelavant of the above, would we still need @jupyterlab/builder too or could we drop it?

[Darshan808]

Is it really expected? Or should jupyter-builder ship with @jupyter/builder assets already?

The current flow is to invoke the Python script via jupyter labextension build ., which internally calls the build script from:

extension/node_modules/@jupyterlab/builder

The new flow changes this to invoking jupyter-builder build ., which calls the build script from:

extension/node_modules/@jupyter/builder

I think it is possible to ship the Python package with the TypeScript build scripts bundled, and invoke them directly. This would remove the need for extensions to depend on an npm package and look into their own node_modules for the build script.

This is worth discussing further, since it impacts the overall build flow and I think it has its own pros and cons.

Would we still need @jupyterlab/builder too or could we drop it?

The end goal is to completely drop @jupyterlab/builder.

However, at the moment we still need it. The reason is that the jupyter_builder Python package currently performs compatibility checks between core-meta and the extension by comparing versions of @jupyterlab/builder in both.

Since @jupyter/builder has not yet been integrated into the JupyterLab core, it is not present in core-meta. Therefore, the compatibility check cannot yet be updated to use @jupyter/builder.

Once the core adopts @jupyter/builder, we can update the compatibility logic to compare versions of @jupyter/builder instead.

Given this, this PR will likely need to remain open for a bit longer and I think it can only be merged after the core fully adopts @jupyter/builder.

[krassowski]

This is worth discussing further, since it impacts the overall build flow and I think it has its own pros and cons.

Agreed, let's track this in an issue on jupyter-builder repo.

Since @jupyter/builder has not yet been integrated into the JupyterLab core, it is not present in core-meta. Therefore, the compatibility check cannot yet be updated to use @jupyter/builder.

Once the core adopts @jupyter/builder, we can update the compatibility logic to compare versions of @jupyter/builder instead.

Given this, this PR will likely need to remain open for a bit longer and I think it can only be merged after the core fully adopts @jupyter/builder.

Is it only because we would have both @jupyter/builder and @jupyterlab/builder? If so we could still undraft this PR and make dropping @jupyterlab/builder a follow-up task, but it would just be a bit confusing for users I guess to have both.

In any case, it looks like we should focus on migrating JupyterLab core to new builder now.