Skip to content

feat: support GCS repository for snapshot operations #766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
gatici merged 22 commits into from
Jan 14, 2026
Merged

feat: support GCS repository for snapshot operations #766

gatici merged 22 commits into from
Jan 14, 2026

Conversation

@gatici
Copy link
Contributor

@gatici gatici commented Dec 8, 2025 •
edited
Loading

Description of issue or feature:

This PR allows Opensearch to use with Google Cloud storage to store snapshots or restore them.
With this change, operators can use a gcs-integrator relation to:

  • create backups (snapshots) into a GCS bucket
  • list existing backups
  • restore backups from GCS
    in the same way they already do with S3 and Azure.

Solution:

GCS Integration

  • Introduced a gcs relation endpoint (GCS_RELATION) and integrated it into the snapshots flow alongside the existing S3 and Azure backends.
  • Extended OpenSearchSnapshotsManager to support ObjectStorageType.GCS:
    • build the repository settings for GCS based on the relation data.
    • call ensure_repository + verify_repository(ObjectStorageType.GCS) to validate connectivity and permissions.
    • reused the existing pre-check logic (leader checks, health, running operations) so GCS behaves consistently with S3/Azure.

How was this change tested?

  • Manually
  • Unit tests
  • Integration tests

Checklist

  • I have added or updated any relevant documentation.
  • I have cleaned any remaining cloud resources from my accounts.

@gatici gatici changed the title feat: add gcs integration for snapshots feat: support GCS repository for snapshot operations Dec 8, 2025
skourta
skourta reviewed Jan 6, 2026
View reviewed changes
Copy link
Contributor

@skourta skourta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @gatici! I left some comments

tests/integration/ha/test_backups.py

logger.info("Setting up configuration for gcs-integrator charm...")
await ops_test.model.applications[GCS_INTEGRATOR].set_config(full_cfg)
await ops_test.model.wait_for_idle(apps=[GCS_INTEGRATOR], timeout=TIMEOUT)
Copy link
Contributor

@skourta skourta Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use wait_until

Copy link
Contributor Author

@gatici gatici Jan 13, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wait_until expects an app status and if we do not provide, it is counted as active. Here we do not want to check the app status as this is a generic helper. We only want to check if juju finishes all work and model reaches a stable state. The status checks are done in the test cases.

akram09
akram09 reviewed Jan 6, 2026
View reviewed changes
Copy link
Contributor

@akram09 akram09 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gatici !

gatici added 18 commits January 13, 2026 16:12
@gatici
feat: add gcs integration for snapshots
16a3aac 
Signed-off-by: gatici <[email protected]>

# Conflicts:
#	lib/charms/opensearch/v0/opensearch_snapshots.py
@gatici
fix: update poetry.lock
10ddcdb 
Signed-off-by: gatici <[email protected]>
@gatici
chore: add integration tests and improve gcs integration logic
f2a32dc 
Signed-off-by: gatici <[email protected]>
@gatici
tests: add spread files for gcs tests
493c4e2 
Signed-off-by: gatici <[email protected]>
@gatici
fix: gcs service account json storage issues
b15df56 
Signed-off-by: gatici <[email protected]>
@gatici
chore: add lost changes in relation_peer_cluster during rebase
aa2c01f 
Signed-off-by: gatici <[email protected]>
@gatici
chore: process gcs credentials when secret_changed
8c7443d 
Signed-off-by: gatici <[email protected]>
@gatici
fix: _gcs_credentials method
e39e22f 
Signed-off-by: gatici <[email protected]>
@gatici
chore: improve pydantic validation
e2f988e 
Signed-off-by: gatici <[email protected]>
@gatici
chore: replace strings with enums
29c0c98 
Signed-off-by: gatici <[email protected]>
@gatici
arrange GCSRelData class and create the service_account.json under sn…
e39d643 
…ap_daemon home

Signed-off-by: gatici <[email protected]>
@gatici
chore: gcs_secret key can be in base64 encoded format
f9be008 
Signed-off-by: gatici <[email protected]>
@gatici
chore: perform base64 decoding in GcsRelDataCredentials class
1cd67ae 
Signed-off-by: gatici <[email protected]>
@gatici
fix: use valid secret format for gcs credentials
b212cc5 
Signed-off-by: gatici <[email protected]>
@gatici
chore: arrange bad credentials
86dac2a 
Signed-off-by: gatici <[email protected]>
@gatici
chore: fix integration tests for gcs large deployment misconfiguration
12b6b16 
Signed-off-by: gatici <[email protected]>
@gatici
chore: simplify verify_gcs_credentials method
07cf0a6 
Signed-off-by: gatici <[email protected]>
@gatici
chore: improve definitions, no change in the behaviour
9951068 
Signed-off-by: gatici <[email protected]>
@gatici gatici requested review from akram09 and skourta January 13, 2026 13:25
Mehdi-Bendriss
Mehdi-Bendriss reviewed Jan 13, 2026
View reviewed changes
Copy link
Contributor

@Mehdi-Bendriss Mehdi-Bendriss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Gulsum, great work! We're already seeing the result of the refactor :)

lib/charms/opensearch/v0/helper_charm.py Outdated

def write_gcs_service_account_json(
secret_key: str,
dst: str = "/var/snap/opensearch/common/home/snap_daemon/service_account.json",
Copy link
Contributor

@Mehdi-Bendriss Mehdi-Bendriss Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this and next function as a broad helpers, as we're ultimately just writing/deleting a text file (which is also not k8s compatible), feel free to use opensearch.write_file as those will ultimately be moved to the Workload class

@gatici gatici requested a review from Mehdi-Bendriss January 13, 2026 19:41
@gatici
fix: typo
1c02e18 
Signed-off-by: gatici <[email protected]>
Mehdi-Bendriss
Mehdi-Bendriss previously approved these changes Jan 14, 2026
View reviewed changes
Copy link
Contributor

@Mehdi-Bendriss Mehdi-Bendriss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Gulsum! I do have one question

skourta
skourta previously approved these changes Jan 14, 2026
View reviewed changes
Copy link
Contributor

@skourta skourta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the hard work @gatici!

@gatici gatici dismissed stale reviews from skourta and Mehdi-Bendriss via bf83b52 January 14, 2026 14:01
@gatici gatici merged commit 64adb1e into 2/edge Jan 14, 2026
156 of 158 checks passed
@gatici gatici deleted the gcs-integration branch January 14, 2026 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@skourta skourta skourta approved these changes

@Mehdi-Bendriss Mehdi-Bendriss Mehdi-Bendriss approved these changes

@smethnani smethnani Awaiting requested review from smethnani

@akram09 akram09 Awaiting requested review from akram09

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@gatici @akram09 @skourta @Mehdi-Bendriss