Expose a package "loader" functionality

[jsoriano]

Package spec currently exposes validation functionality through the validator package. Create another package, loader, with package loading functionality, to read the contents of packages in a best-effort basis, without requiring strict validation.

Ideally this loader could be mostly auto-generated from the package spec itself. It could access content in a lazy way, so parsing is only done for what is needed for each application. It should be tolerant to failures, so content can be accessed even if the package is partially invalid or incomplete. This loader would abstract handling of different format versions or package types.

Partial package loading functionality is required in several places, each of them is currently maintaining their specific implementation:

• Indexers read packages. Package registry reads packages to index them. Package storage v2 indexer will also need to read packages to generate indexes.
• Kibana reads packages. Exposing information from packages was requested as feedback for the wasm poc.
• elastic-package reads packages manifests and fields files.
• Documentation exporters read the manifest and documentation files.

This loader wouldn't mean that we discourage third-party implementations of package consumers, the specification would still be there the same way as it is now. In that sense this loader could be seen as a sample implementation.

[mtojek]

Maybe we can go further and generate the model from spec? It will prevent having at least 3 partially implemented models in our codebase.

[jsoriano]

Yes, something like this is what I mean with auto-generating from the spec.