Allow users to specify bevy components in external tools and make external tools compatable by supporting Bevy gltf custom property

[Igor-dvr]

What problem does this solve or what need does it fill?

There are various formats like GLTF, FBX, OpenUSD, etc that allow to transfer meshes, materials, animations, etc. Various tools may need to transfer some additional data to cover their specific needs. For such purposes these formats provide custom properties that can be appended to objects, meshes, materials.. and extend their description. Currently we don't have a builtin solution to import Bevy components from external tools. Custom properties can be used to transfer Bevy components.

Custom properties is commonly used feature and editors like Blender, Maya, Houdini, etc provide interface where custom properties can be specified.

Benefits of supporting Bevy components custom property:

• Users can specify Bevy components in external tool and after importing an asset have all initial components already in place.
• External tools like Blenvy/Skein that currently implement their own hacky Bevy components importers can migrate to Bevy solution and extend it.
• Maya/Houdini/Blender/Skein/Blenvy or any tools that allow to specify (Bevy) custom property becomes compatible. This allows to migrate from one tool to another without loosing specified Bevy components.
• Its much easier to implement external tool/plugin that allows specifying Bevy components.
• ...

What solution would you like?

We already support GLTF custom properties. What is needed is to extend GLTF importer and parse Bevy defined custom property responsible for transferring components. When importing GLTF asset and iterating over asset objects, materials, meshes..the importer should parse Bevy custom property if present and insert parsed component.

Custom property could look something like this:
"bevy_components { MarkerComponent1, ComponentWithFields1 {.. }, ComponentWithFields2 {.. }, ..}"
or we can use an array or something else.

Alternatively our own Bevy GLTF extension can be implemented instead of using custom properties. Or we can implement both.

[alice-i-cecile]

I'm generally interested in this, but would like to see a more full design. @ChristopherBiscardi, I care a lot about your opinion here.

[ChristopherBiscardi]

I've had the idea of a "bevy gltf extension for components" in the back of my head for awhile while working on skein. This is one of the reasons I keep skein's implementation very compact without extra features.

I think the format skein produces is pretty close to optimal for reflection. This is basically exactly what reflection needs, and is packed into a vec (instead of say, a single object, which one could say is an aesthetic choice, but enables workflows like creating multiple components of the same type while debugging).

Here's a concrete example of the gltf data so people can get on the same page.
The data format can be parsed directly into a serde_json::Value and sent through a ReflectDeserializer.

{
  "doubleSided": true,
  "extras": {
    "skein": [
      {
        "replace_material::UseForceFieldMaterial": {}
      },
      {
        "bevy_light::NotShadowCaster": {}
      },
      {
        "bevy_light::NotShadowReceiver": {}
      }
    ]
  },
  "name": "ForceField",
  "pbrMetallicRoughness": {
    "baseColorFactor": [
      0.800000011920929,
      0.800000011920929,
      0.800000011920929,
      1
    ],
    "metallicFactor": 0,
    "roughnessFactor": 0.5
  }
}

A struct Character { name: String } looks like this.

{
  "event_ordering::Character": {
    "name": "Hollow Knight"
  }
}

Given the above data and a serde_json::Value value, the implementation is amazingly small. Skein expands this code to include error messaging, etc but the fundamental code is these four lines in an Observer.

let type_registry = type_registry.read();

// deserialize
let reflect_deserializer = ReflectDeserializer::new(&type_registry);
let reflect_value = reflect_deserializer.deserialize(json_component)?;

commands.entity(entity).insert_reflect(reflect_value);

So this is in general not a very hacky solution at all, it relies on Bevy's core features (Component lifecycle events and reflection).

Insertion location

Doing this work in an observer is fine and is the most flexible solution currently, but there are two other places Components could be inserted.

0. Main World Observer (as is indicated above)
1. To the Scene World directly after the Gltf Scenes are created (post-asset loader, in userspace)
2. In the glTF loader directly as nodes, etc are being created and spawned.

Data Storage location

The data is currently stored in GltfExtras, but this is a choice made on practicality. Bevy has great support for GltfExtras, including allowing component lifecycle events, and comparatively poor support for gltf extensions, which require code modifications to bevy's glTF loader to make use of.
This means users who want to work with sub-assets of the glTF file can grab those assets and retain the ability to use the components that were externally specified.

sub-asset usage can enable those components like this as a result;

let handle = gltf.named_meshes.get("Sphere").unwrap();
let gltf_mesh = gltf_meshes.get(handle).unwrap();

commands.spawn((
    Mesh3d(gltf_mesh.primitives[0].mesh.clone()),
    MeshMaterial3d(materials.force_field.clone()),
    gltf_mesh.primitives[0]
        .extras
        .clone()
        .unwrap_or(GltfExtras::default()),
));

A Bevy Component Extension

So if this were going to be a real upstreamed thing, I think there's a couple of hurdles on the path to making this a reality:

• Skein uses On<Add, GltfExtras> observers to handle component insertion. An official implementation probably wants to do this in the gltf loader or as a post-process on the scene world. This is also what Skein probably needs to do to support relationships anyway.
• gltf extension data is associated with nodes (as well as allowing global data) but Bevy doesn't really expose this currently. So we might want to, or we might want to officially commit to using gltfextras. I think gltfextras is the "worse" choice from a spec compliance perspective, and currently the better choice for practical support today, so we'd close the gap for the extension.
• Its worth compiling a list of other software that would want to use a similar approach, or even could use a similar approach. For software that can't do it directly with an addon (ex: like Blender can), how can the data be defined and published appropriately into the gltf file. I'd like to see at least one other piece of software get an extension targeting the format described above.
• Relationship support

Relationships and Ids

Bevy has the concept of Relationship components (or just components that store Entity).
glTF does not have the concept of ids (unique ids are not generally considered easy to implement, as evidenced by KhronosGroup/glTF#2337 ), and Names are not guaranteed to be unique (but could be a soft solution).
This makes supporting Relationships a bit awkward;
Other scene representations specify Entity ids ahead of time (scene.scn.ron in our examples), but the glTF loader creates them while loading, ad-hoc.
This means all Relationship components effectively need to be a post-processing insertion and there needs to be some way to "target" other nodes.
I'm experimenting here but don't have a concrete solution to suggest yet.

Other

Components can include Handles which can not be used from external software since Handles don't exist. FogVolume for example, includes an Option<Handle> which requires code to automatically set this to None currently (or implement some workaround to create and specify the Handle when loading).

struct FogVolume {
    pub fog_color: Color,
    pub density_factor: f32,
    pub density_texture: Option<Handle<Image>>,
    pub density_texture_offset: Vec3,
    pub absorption: f32,
    pub scattering: f32,
    pub scattering_asymmetry: f32,
    pub light_tint: Color,
    pub light_intensity: f32,
}

Somewhat tangential, but the reflection data also changes between Rust versions. This isn't really a Bevy concern yet imo as there is no LTS version, but its notable that between 0.16 and 0.17 components like NotShadowCaster moved from bevy::pbr::NotShadowCaster to bevy::light::NotShadowCaster which is a breaking change from a reflection point of view and would cause the need for the DCC extensions to provide migration tools or users to re-insert those components with the appropriate paths.

So yeah, I've been quietly working towards this with the goal of proposing it when I thought it was ready. While the core component model I describe above feels very solid, it excludes Relationships and there's some more experimentation to do to figure out the right solutions for Relationships and figuring out if extensions is where we want to put the data long-term.

I do know that other people have taken the code from skein and used it successfully with other software, so there is precedent for the format described and general rust-side approach being "good enough".

[ChristopherBiscardi]

it would be great to get a suggestion for another piece of software to support from someone who uses not-Blender that supports scripted addons/gltf export extensions/etc that I could play with supporting in skein.

[ChristopherBiscardi]

Also, the data format described above currently works well with, for example Trenchbroom, to set the data in Blender, place items in a level with Trenchbroom, and get the components instantiated in Bevy. This is pretty reasonable precedent that this would support functionality like when the future Bevy Editor or Asset processing to turn bsn into multiple assets exists.

[Igor-dvr]

By "hacky" I mean that probably importer should be located in asset loader if we impl this in Bevy directly.

Data storage location. Looks like there are 3 main options:

• Custom properties (GLTF extras). Blender provides native support for this. Blenvy/Skein can provide additional features like extended interface and fetching a list of available components from Bevy, etc. Maya supports GLTF extras via plugin https://doc.babylonjs.com/features/featuresDeepDive/Exporters/Maya_to_glTF . Houdini "custom attributes" support mentioned here https://www.sidefx.com/docs/houdini/nodes/out/gltf.html . 3DS MAX "custom attributes" support mentioned here https://doc.babylonjs.com/features/featuresDeepDive/Exporters/3DSMax_to_glTF .
  Also, GLTF and OpenUSD custom properties are compatible (at least partially) and assets with custom properties can be converted from Gltf to OpenUSD using Blender. User can work in a tool that only supports OpenUSD output then import OpenUSD in Blender and export GLTF that will be used in Bevy.
  If a user used Blenvy/Skein or other custom plugin to specify components and the plugin stopped being supported user can use Blender native panels or migrate to something else.
• Gltf extension. If we impl our own GLTF extension we'll need to impl external plugins that import/export this extension. Conversion from GLTF to other format will loose components if we don't provide support for the target format.
• Create our own format to transfer components? This will require even more work on the side of external tools.

Custom properties seems like a simple, interoperable solution.

I currently don't plan to implement this but I can review if someone implements.
We can implement the feature as experimental first and see if it needs improvements.

[Igor-dvr]

When specifying a struct (component) name in a custom property do we need to use a full struct name? If we can use a short struct name like NotShadowCaster instead of full bevy::light::NotShadowCaster users will not have to worry about migration if component is moved in code. But code can have structs with the same name in different modules and full name is needed in such situations.
Maybe we can support short and full struct names in custom property. If there is only one struct with specific name we can use short name. If there are multiple structs with the same name we can use long names. Not sure how hard is to impl this.

[ChristopherBiscardi]

When specifying a struct (component) name in a custom property do we need to use a full struct name?

Yes we do. Reflection deserialization requires the full path to know the type and even if it didn't its trivial to have multiple components with the same name and not know which to deserialize to, which would end up in a situation where adding a component to your app or adding a third party dependency can break your gltf component data.

[ChristopherBiscardi]

Create our own format to transfer components? This will require even more work on the side of external tools.

Creating a custom file format or additional sidecar file, etc should be an absolutely last resort, if even acceptable. I would even go so far as to say that programs that can't output in the correct format inside of a gltf file and need a sidecar file should also have a postprocess step to insert that data into gltf (extras || extension) after exporting, and the format bevy receives should stay simple.

Note

I do think there's value in a BSN exporter from these programs, which is a different but related project that we can't really experiment with until the .bsn asset format exists

extras vs extensions for data is something I want to do an implementation for each to compare before declaring either one better. Like I said in a previous comment, the choice to use extras was a practical choice to get something working outside of upstream without requiring replacing the loader. Extensions is, afaict, the "right place" according to the spec to put data like this. extras are for "application specific data", which implies to me that it is a place for App-specific data, not something as general as reflected component data across all bevy apps.

And upstreaming support for gltf loading lifecycle events to allow users to process extensions as nodes are processed or something similar is a possibility. Right now there's no way for anyone to implement an extension outside of upstream, but the "parse" function is standard amongst the various extensions in use

Any gltf object can have extension-related data, and it also enable things like storing "global" data and additional functionality like accessors.

Gltf extension. If we impl our own GLTF extension we'll need to impl external plugins that import/export this extension.

Realistically someone will need to do this anyway. The format is possible to hand-write, which provides insulation against an addon going out of date, but very few people are going to want to do that as a default.

Conversion from GLTF to other format will loose components if we don't provide support for the target format.

This will happen anyway. Not all formats have easily accessible extension/extras locations and gltf can represent quite a few different kinds of data.

[ChristopherBiscardi]

Wrote this all up and posted it here while I work through the rest of the details: https://www.christopherbiscardi.com/bevy-components-and-gltf