Investigate leveraging upstream EIP catalog

[igarashitm]

Currently Kaoto uses camel-component-metadata as an EIP catalog
https://github.com/KaotoIO/camel-component-metadata

However the Camel upstream EIP catalog and YAML DSL schema might already satisfy our needs. In fact they have what we're currently missing, for example the list of dataformats and languages. Investigate to leverage it in Kaoto, and possibly discontinue camel-component-metadata above.

Findings

Kaoto specific properties in camel-component-metadata

EIP/EIP-BRANCH

According to Maria this is used in earlier Kaoto but no longer important. Whether the steps could have branch or not could be identified through schema, see the following section.

Icons

These are not in Camel upstream artifacts and Kaoto needs to keep. e.g. migrate as a static resource in kaoto-backend/camel-support

minBranches/maxBranches

These are also Kaoto specific properties. Either put in a static resource together with Icons or attach programmatically.

What's the available steps?

Except from step, which is very special one, all the steps available in the Camel Route are in org.apache.camel.model.ProcessorDefinition in the YAML DSL schema here
https://github.com/apache/camel/blob/camel-3.20.x/dsl/camel-yaml-dsl/camel-yaml-dsl/src/generated/resources/schema/camel-yaml-dsl.json#L7C8-L7C50
And for each steps, there's a corresponding model catalog entry. For example aggregate in the schema
https://github.com/apache/camel/blob/camel-3.20.x/dsl/camel-yaml-dsl/camel-yaml-dsl/src/generated/resources/schema/camel-yaml-dsl.json#L12
maps to the aggregate.json model catalog entry
https://github.com/apache/camel/blob/camel-3.20.x/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/models/aggregate.json
They are linked together with the model class name (e.g. org.apache.camel.model.AggregateDefinition) as a key. We need both information for Kaoto, while the YAML DSL schema defines the structure of the flow, the model catalog provides detailed metadata for each steps.

Note: KameletBinding / Pipe binding steps

These things specified through source, steps and sink are specially restricted on what could be specified. Need further investigation. On the other hand the side routes defined in KameletBinding/Pipe spec.integration.flows are just regular Camel Route. cf. kaoto-archive/kaoto-ui#2155

Note: Restricting available steps through YAML DSL schema

Camel has camel-yaml-dsl-maven-plugin to generate the YAML DSL schema
e.g. https://github.com/igarashitm/cos-fleet-catalog-camel/commit/6de408ed7e35cec22f3af8c4f7b632fadef05914#diff-1658422f830fa328b8f1e0e616da0902c8810a180a62479e1ad9f18f99ce51f4R356
and it has an option to exclude specific definition, called bannedDefinition
e.g. https://github.com/igarashitm/cos-fleet-catalog-camel/commit/6de408ed7e35cec22f3af8c4f7b632fadef05914#diff-1658422f830fa328b8f1e0e616da0902c8810a180a62479e1ad9f18f99ce51f4R365
We can govern what steps are available with this feature. Also we can do same for dataformats/languages.

What's the available dataformats/languages?

This is relatively simple, we can either

1. Look at the dataformats/languages catalog
   a. dataformats - https://github.com/apache/camel/tree/camel-3.20.x/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/dataformats
   b. languages - https://github.com/apache/camel/tree/camel-3.20.x/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/languages
2. Or look at the YAML DSL schema
   a. DataFormatsDefinition - https://github.com/apache/camel/blob/camel-3.20.x/dsl/camel-yaml-dsl/camel-yaml-dsl/src/generated/resources/schema/camel-yaml-dsl.json#L4761
   b. ExpressionDefinition - https://github.com/apache/camel/blob/camel-3.20.x/dsl/camel-yaml-dsl/camel-yaml-dsl/src/generated/resources/schema/camel-yaml-dsl.json#L6318

Which are the branching steps?

The step which has an array of "#/items/definitions/org.apache.camel.model.ProcessorDefinition as a property, is a branching step. For example aggregate has steps
https://github.com/apache/camel/blob/camel-3.20.x/dsl/camel-yaml-dsl/camel-yaml-dsl/src/generated/resources/schema/camel-yaml-dsl.json#L492-L497

Note: An array of ProcessorDefinition could be either serial or parallel

For example, the steps under when is supposed to be serial, sequentially executed. On the other hand, the steps under load-balance is supposed to be parallel, laying flat, just a collection of single step execution that could be invoked in parallel. These are semantically different, but it doesn't clearly appear in the schema or catalog.
cf. #763 , kaoto-archive/kaoto-ui#2094

What's the javaType in the Camel catalog?

There are roughly 2 types for this javaType

1. One is as we can see in aggregate, this javaType refers to a Camel model class name. The model catalog and the definition in the YAML DSL schema are linked together with this javaType as a key. Same thing applies for some of component/step parameters that is not primitive.
2. The other is, "not the camel model class". For example connectionFactory on AMQP component https://github.com/apache/camel/blob/camel-3.20.x/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/amqp.json#L26
   Usually this is supposed to be configured with referencing a bean definition. In this case the javaType here could be used to filter which bean could be applied for this parameter.

Do we inline everything in Kaoto step catalog?

At least we'd want to keep dataformats/languages separated from step catalog since it repeats again and again. For example the backend can provide the separate endpoint for returning schema/catalog metadata for dataformats and languages. Then frontend would query it on initialization and keep them in a store. When frontend render the step configuration, it also refers to those dataformats and languages metadata when a step parameter refers to them through javaType.