Feature/voxel annotation by briossant · Pull Request #858 · google/neuroglancer

briossant · 2025-10-30T16:22:42Z

This Draft Pull Request introduces an interactive voxel annotation feature, allowing users to perform manual segmentation by painting directly onto volumetric layers. This implementation is based on the proposal in Issue #851 and incorporates the feedback from @jbms.

Here is a live demo to try the feature, watch out, there is persistent storage, so your annotations will be saved and will override the ones already present: OPEN DEMO VIEWER

Key Changes & Architectural Overview

Following the discussion, this implementation has been significantly revised from the initial prototype:

Integration with Existing Layers: Instead of a new vox layer type, the voxel editing functionality is now integrated directly into ImageUserLayer and SegmentationUserLayer via a UserLayerWithVoxelEditingMixin. This mixin adds a new "Draw" tab in the UI.

New Tool System: The Brush and Flood Fill tools are implemented as toggleable LayerTools, while the Picker tool is a one-shot tool. All integrate with Neuroglancer's new tool system. The drawing action is bound to Ctrl + Left Click.
Optimistic Preview for Compressed Chunks: To provide immediate visual feedback and solve the performance problem with compressed chunks, edits are now rendered through an optimistic preview layer.

When a user paints, edits are first applied to an InMemoryVolumeChunkSource.
This preview source is rendered by a second instance of the layer's primary RenderLayer (e.g., ImageRenderLayer or SegmentationRenderLayer). This ensures the preview perfectly matches the user's existing shader and display settings.
The base data chunk is not modified on the frontend, avoiding the need to decompress/recompress it.

Data-flow

sequenceDiagram
participant User
participant Tool as VoxelBrushTool
participant ControllerFE as VoxelEditController (FE)
participant EditSourceFE as OverlayChunkSource (FE)
participant BaseSourceFE as VolumeChunkSource (FE)
participant ControllerBE as VoxelEditController (BE)
participant BaseSourceBE as VolumeChunkSource (BE)

    User->>Tool: Mouse Down/Drag
    Tool->>ControllerFE: paintBrushWithShape(mouse, ...)
    ControllerFE->>ControllerFE: Calculates affected voxels and chunks

    ControllerFE->>EditSourceFE: applyLocalEdits(chunkKeys, ...)
    activate EditSourceFE
    EditSourceFE->>EditSourceFE: Modifies its own in-memory chunk data
    note over EditSourceFE: This chunk's texture is re-uploaded to the GPU
    deactivate EditSourceFE

    ControllerFE->>ControllerBE: commitEdits(edits, ...) [RPC]

    activate ControllerBE
    ControllerBE->>ControllerBE: Debounces and batches edits
    ControllerBE->>BaseSourceBE: applyEdits(chunkKeys, ...)
    activate BaseSourceBE
    BaseSourceBE-->>ControllerBE: Returns VoxelChange (for undo stack)
    deactivate BaseSourceBE
    ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC]
    activate ControllerFE
    ControllerFE->>BaseSourceFE: invalidateChunks(chunkKeys)
    note over BaseSourceFE: BaseSourceFE re-fetches chunk with the now-permanent edit.
    ControllerFE->>EditSourceFE: clearOptimisticChunk(chunkKeys)
    deactivate ControllerFE

    ControllerBE->>ControllerBE: Pushes change to Undo Stack & enqueues for downsampling
    deactivate ControllerBE

    loop Downsampling & Reload Cascade
        ControllerBE->>ControllerBE: downsampleStep(chunkKeys)
        ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC]
        activate ControllerFE
        ControllerFE->>BaseSourceFE: invalidateChunks(chunkKeys)
        note over BaseSourceFE: BaseSourceFE re-fetches chunk with the now-permanent edit.
        ControllerFE->>EditSourceFE: clearOptimisticChunk(chunkKeys)
        deactivate ControllerFE
    end

Writable source selection Aside every activated volume sub-sources of a writable datasource, an additional checkbox lets the user mark the sub-source as writable, then neuroglancer will try to write in it.

~~5. Dataset creation To complete Neuroglancer's writing capabilities, a dataset metadata creation/initialization feature was introduced.~~

~~The workflow is triggered when a user provides a URL to a data source that does not resolve:~~

~~Neuroglancer recognizes the potential intent to create a new dataset and prompts the user:~~

~~Finally, the user is able to access dataset creation form:~~

Data sources & Kvstores

Currently, there is a very limited set of supported data sources and kvstores, which are:

datasources:
- zarr v2 and v3 with codecs: raw, gzip, blosc
kvstores:
- s3+http(s): can be used with a local s3 bucket (e.g. minio) or with anonymous s3 urls
- ~~opfs: in-browser storage, also used for local development at some point, the relevancy can be discussed.~~
- ~~ssa+https: a kvstore linked to an in development project, which is a stateless (thanks to OAuth 2.0) worker providing signed urls to read/write in s3 stores~~

Limitations

Only the rank 3 volumes are supported
Float32 dataset are not supported
Unaligned chunk hierarchy/resolutions (e.g. child chunks that may have multiple parents) is not supported

Open Questions & Future Work

This PR focuses on establishing the core architecture. Several larger topics from the original discussion are noted here as future work:

Efficient Low-Resolution Drawing: As discussed, efficient, multi-resolution drawing with upsampling is a complex challenge that requires a new data format.
3D Drawing Tools: As suggested by @fcollman, 3D-specific tools like interpolation between slices are out of scope for this initial PR but could be a valuable direction for future work.

Checklist

Completed the todo list found in src/voxel_annotations/TODOs.md
~~[ ] Added support to more (every?) datasources and kvstores~~
Signed the CLA.

Edits

updated zarr support
added "5. Dataset creation" section
strikethrough what's no longer part of this PR
added the Limitations section
added live demo

google-cla · 2025-10-30T16:22:49Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

jbms · 2025-11-03T22:55:18Z

Can you complete the CLA?

src/layer/voxel_annotation/index.ts

src/layer/vox/controls.ts

jbms · 2025-11-03T23:04:17Z

The brush hover outline (circle where the mouse pointer is) seems to go away in some cases when changing the zoom level.

briossant · 2025-11-10T09:58:45Z

~~I need to rewrite history, my commits are signed with the wrong email, I will open a new PR I made a mistake with this force push~~ nevermind its fixed

- Introduced a new dummy `MultiscaleVolumeChunkSource`. - Added `VoxelAnnotationRenderLayer` for voxel annotation rendering. - Implemented `VoxUserLayer` with dummy data source and rendering. - Added tools and logs for voxel layer interactions and debugging. - Documented voxel annotation specification and implementation details.

- Added a backend `VoxDummyChunkSource` that generates a checkerboard pattern for voxel annotations. - Implemented frontend `VoxDummyChunkSource` with RPC pairing to the backend. - Updated documentation with details on chunk source architecture and implementation.

…t seems there is some fighting.

…s to corruped the chunk after the usage of the tool. Added a front end buffer which is the only drawing storage for now. Added user settings to set the voxel_annotation layer scale and bounds. Added a second empty source to DummyMultiscaleVolumeChunkSource to prevent crashs when zoomed out too much

…lobal one (there where a missing convertion) ; add a primitive brush tool

…umeChunkSource and update related imports

…panded brush settings

…ation, and improved backend edit handling

…map options and UI settings

…r remote workflows, label creation, and new drawing tools

…table source is present

…ization and add benchmarks to prouve the performance gain

…is undefined, added a benchmark to see the performance gain

…allocation within the loops and breaking down the maths

…operly calculate the stamina

…t offset based on brush radius

use BackendVoxelAccessor for the downsmapling, keep the created BackendVoxelAccessor in cache and set a max cache size in BackendVoxelAccessor.

…E_DISPLYING_3_DISKS as the SPHERE), optimize the frontend brush by flattening vectors

…ft` is held

…l when brush tool is active

…rently 50ms) and synced, this mean that it will temporarly override pixel of chunks not loaded in memory. This feature may be discussed as the caused artifacts may be judged too strong.

… call per stroke to reduce the amount of applyLocalEdits calls

… and fix the integration tests

…last commit

briossant · 2026-01-29T18:17:21Z

Update on the latest work:

I've addressed the feedback from @chrisj as detailed in my latest message.
I added the "compute load indicator", which I now call the stamina bar. It appears and gets emptied when drawing, and fills up as the backend works. If the stamina bar is visible and you reload the page, drawings may be lost. The stamina computation, found in src/voxel_annotation/base.ts, is subject to change.
Optimization work has been done to the frontend and backend drawing computations and the downsampling.
I added a fast preview for the flood fill, though it is debatable whether we should keep it. It has a limited range and may sometimes flood unwanted chunks if they are not loaded in memory.
For the preview of the sphere I kept the 3 disks.

DEMO LINK

…lution dataset)

briossant · 2026-01-30T13:12:43Z

Using a dataset with a single resolution, we can use the voxel annotation without the downsampling, reducing the complexity. Here is an example.

davidackerman · 2026-02-19T20:38:22Z

I just wanted to say this feature is awesome and has been a dream feature for many of us! I think it would be very useful, even if no more changes were made. Perhaps if it is in flux, it could be added as an experimental feature similar to how Volume Rendering was introduced?

For example, we have made a tool that uses python Neuroglancer to do real time, on the fly inference of datasets as one browses around the dataset: https://github.com/janelia-cellmap/cellmap-flow. Using this voxel annotation branch, we have also added the ability to do human in the loop finetuning of our models, akin to ilastik (see below). In the movie, we are doing live preprocessing, inference and postprocessing of one of our models that is doing poorly on nucleus segmentations. We then do finetuning using voxel annotations and get better segementation results, all served up on the fly.

If this was merged into main, it would make maintenance on our end much more straightforward.

CellmapFlowLowRes.mp4

briossant · 2026-03-13T19:28:41Z

I just wanted to say this feature is awesome and has been a dream feature for many of us! I think it would be very useful, even if no more changes were made. Perhaps if it is in flux, it could be added as an experimental feature similar to how Volume Rendering was introduced?

For example, we have made a tool that uses python Neuroglancer to do real time, on the fly inference of datasets as one browses around the dataset: https://github.com/janelia-cellmap/cellmap-flow. Using this voxel annotation branch, we have also added the ability to do human in the loop finetuning of our models, akin to ilastik (see below). In the movie, we are doing live preprocessing, inference and postprocessing of one of our models that is doing poorly on nucleus segmentations. We then do finetuning using voxel annotations and get better segementation results, all served up on the fly.

If this was merged into main, it would make maintenance on our end much more straightforward.

Cool to see the feature already used! I completely agree with you. I've been a bit quiet over the last month since I think all the requested changes and feedback have been addressed, and I would avoid extending the scope of this PR further. Getting this merged as an experimental feature would be fantastic and make it much easier for everyone to start building on and give feedback.

chrisj · 2026-03-13T19:42:06Z

@briossant I've been meaning to spend some time to review the code, it's substantial! I'll try to get to this soon

seankmartin

Thanks for all the work on this @briossant! I'm sorry I'm late to the party in offering any feedback. This PR was (and still is) very extensive and I was struggling to get around to looking at in detail beyond trying the demo a few times to get a feel for the functionality (thank you for making the demo).

Generally it looks pretty good to me as an experimental feature. Very happy to chat more on any of the feedback here, and thanks again

seankmartin · 2026-04-01T10:31:56Z