feat: Data Interfaces V1

[Gu1nness]

Implementation of Data Interfaces V1:

• Provides repositories, interfaces, helper methods
• Secret encryption/decryption based on typing of fields in models
• Missing a bit of documentation (will be added in the coming days)
• Unit + Integration tests
• Examples in the integration tests + documentation
• Supports data interfaces (basic) + permissions + entity

This V1 supports multiple requests from the requirer, and supports both bulk events (1 event for all the requests at the same time) and single events (one event per request), according to your usecase.

It supports credentials updates from the provider (used by some charms), it supports MTLS from the requirer, and heavily relies on type systems and pydantic!

Example of use

from charms.data_platform_libs.v1.data_interfaces import (
    RequirerCommonModel,
    RequirerDataContractV1,
    ResourceCreatedEvent,
    ResourceEntityCreatedEvent,
    ResourceProviderModel,
    ResourceRequirerEventHandler,
)

class ApplicationCharm(CharmBase):
    # Application charm that connects to database charms.

    def __init__(self, *args):
        super().__init__(*args)

        requests = [
            RequirerCommonModel(
                resource="clientdb",
            ),
            RequirerCommonModel(
                resource="clientbis",
            ),
        ]
        # Define the cluster aliases and one handler for each cluster database created event.
        self.database = ResourceRequirerEventHandler(
            self,
            relation_name="database"
            relations_aliases = ["cluster1", "cluster2"],
            requests=
        )
         self.framework.observe(
             self.database.on.cluster1_resource_created, self._on_cluster1_resource_created
         )
         self.framework.observe(
             self.database.on.cluster2_resource_created, self._on_cluster2_resource_created
         )

     def _on_cluster1_resource_created(self, event: ResourceCreatedEvent) -> None:
         # Handle the created database on the cluster named cluster1

         # Create configuration file for app
         config_file = self._render_app_config_file(
             event.response.username,
             event.response.password,
             event.response.endpoints,
         )
         ...

     def _on_cluster2_resource_created(self, event: ResourceCreatedEvent) -> None:
         # Handle the created database on the cluster named cluster2

         # Create configuration file for app
         config_file = self._render_app_config_file(
             event.response.username,
             event.response.password,
             event.response.endpoints,
         )
         ...

[Gu1nness]

Thanks for the amazing job done here, @Gu1nness!

I started to test it against the PostgreSQL 16 K8s on CI and locally (it's still a work in progress, but already given me more confidence in the upgrade of the library).

I have two questions:

* Is the V1 library supposed to also support Juju 2.9 charms or only Juju 3.x charms?

This is only for Juju 3 charms

* Is this V1 library compatible with older versions of the V0 library? I mean, there is some code like the one from [`3c4f5d4`/lib/charms/data_platform_libs/v0/data_interfaces.py#L2507C9-L2507C64](https://github.com/canonical/data-platform-libs/blob/3c4f5d4df8a66bb09efbe2fd5ee8bbf3bea17d3a/lib/charms/data_platform_libs/v0/data_interfaces.py#L2507C9-L2507C64), which migrates from an old internal secret to the structure we have today. Is it supported?

There's is no backward compatibility with pre-secret charms, so no compatibility with the older versions of the library, this was explicitely requested by Mykola.
I think in your case (postgresql), it means you can use it for Postgresql 16 but probably not for Postgresql 14 because of the juju 2.9 compatibility.

[marceloneppel]

Thanks for the amazing job done here, @Gu1nness!
I started to test it against the PostgreSQL 16 K8s on CI and locally (it's still a work in progress, but already given me more confidence in the upgrade of the library).
I have two questions:

* Is the V1 library supposed to also support Juju 2.9 charms or only Juju 3.x charms?

This is only for Juju 3 charms

* Is this V1 library compatible with older versions of the V0 library? I mean, there is some code like the one from [`3c4f5d4`/lib/charms/data_platform_libs/v0/data_interfaces.py#L2507C9-L2507C64](https://github.com/canonical/data-platform-libs/blob/3c4f5d4df8a66bb09efbe2fd5ee8bbf3bea17d3a/lib/charms/data_platform_libs/v0/data_interfaces.py#L2507C9-L2507C64), which migrates from an old internal secret to the structure we have today. Is it supported?

There's is no backward compatibility with pre-secret charms, so no compatibility with the older versions of the library, this was explicitely requested by Mykola. I think in your case (postgresql), it means you can use it for Postgresql 16 but probably not for Postgresql 14 because of the juju 2.9 compatibility.

Thanks for the clarification.

[deusebio]

+1 here from Cassandra folks! 🥳

We have integrated the library successfully on cassandra. Is there any timeline when this PR is going to be merged or should we just go ahead and merge the Cassandra PR? I suppose we can always update the lib once it is consolidated

[skourta]

+1 here from Cassandra folks! 🥳

We have integrated the library successfully on cassandra. Is there any timeline when this PR is going to be merged or should we just go ahead and merge the Cassandra PR? I suppose we can always update the lib once it is consolidated

@deusebio any reason why you did not migrate the peer relation data handling to v1?

[zmraul]

As we discussed, charm can be removed from this class, as it is not used. This avoids having to send the charm object to data model focused classes.

[Gu1nness]

Updated, hopefully I forgot no use case.

[deusebio]

any reason why you did not migrate the peer relation data handling to v1?

I'd say to keep the PR confined and capacity. My assumption (also looking at the spec DA175) is that data-interfaces were mostly concerned (and critical) for client<>server relations. Of course we could also apply them to peer, but that's nice to have for us right now. We have a few peer relations, so this may not be a trivial change. The current task we need to deliver before end of cycle are implementing client relations (the PR attached) and implement HA tests. There is not much time left before the end of the cycle and we would like to wrap those up.

TLDR I believe using a more structured approach for peer can also be useful, but probably to be addressed in the future

[Gu1nness]

@deusebio @zmraul @skourta: I've updated the interfaces following the comments from @zmraul :
All interfaces now have the model: Model in first argument instead of charm: CharmBase.
The data model for those interfaces is now known as data_model instead of model to respect those interfaces.

@zmraul Looks better ?

[reneradoi]

Thank you @Gu1nness for all your effort and your work on this PR! I approve, no blockers from my side.

[zmraul]

@zmraul Looks better ?

Yes, thank you! This is a nice improvement :)

[imanenami]

This is already fabulous for the first release of V1, and we have integrated it successfully with the main Kafka charm, so I'm happy with it. Great effort @Gu1nness!

[skourta]

Thank you, @Gu1nness. Great work! All needed features and helpers to have full migration to v1 are now included.

[Mehdi-Bendriss]

Outstanding work Neha!