Remove Connection dependency from shared secrets backend

[amoghrajesh]

---

Was generative AI tooling used to co-author this PR?

• No

Why?

The shared BaseSecretsBackend was importing Connection classes from airflow-core or airflow-sdk conditionally based on the current process context, creating a dependency that breaks client-server separation in ways. This change removes that dependency by injecting the Connection class per instance at distribution init time.

Both core and sdk have backend loaders: initialize_secrets_backends which now set the Connection class on
each backend instance via _set_connection_class() when instantiating their backends. This ensures each backend uses the correct Connection type (core or sdk) based on where it's created, without the shared base needing to know about either.

Not so ideal, but there could be cases when the secrets backend class is instantiated manually and used, to account for such cases, we have a fallback which sets Connection from respective library / consuming module on its re export of BaseSecretsBackend. ie: _connection_class is stored on the instance (self._connection_class).
The “default” is just the logic that decides what to set when the instance has nothing; it’s not a shared class attribute.

The shared base now has zero imports of core or SDK Connection types, maintaining proper separation between components.

Design Decision and Alternatives

Why per instance instead of class level default?

I considered a class level _default_connection_class attribute, but didn't choose it because:

1. Core imports would overwrite task-sdk defaults when loading backends or vice versa
2. In processes importing both core and task-sdk, whichever set the default last would win, regardless of context
3. All backend instances would share one default state which was mutable, causing interference between backends

Per instance storage avoids all these issues by making each backend independent.

Why not use a factory/callable?

I also considered storing a callable that returns the Connection class, but chose direct class storage over that one cos:

• Simpler implementation
• Same robustness with less complexity

Back compat

• No changes to public API
• Existing backend code continues to work
• Wherever the backend are created, they are initialised at runtime via Backends created via initialize_secrets_backends. So, they automatically get the correct Connection class set on the instances.

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[amoghrajesh]

@potiuk I vaguely remembering being tagged in one such task from you, or maybe I am mistaken but hopefully this was the one!

[amoghrajesh]

Most of the failures in here are handled by: #61728

[o-nikolas]

The tests are still failing, but the code looks reasonable to me. Though, I don't work much on the SDK/Core boundary so take this review with a large grain of salt 😅

[amoghrajesh]

Thanks for the review @o-nikolas! It is dependent on #62211

[amoghrajesh]

Remaining failures to be handled by: #62581

cc @potiuk can you take a look? You might be most interested here

[amoghrajesh]

Finally! Merging this one