Skip to content

Allow list of available xcache servers to be downloaded #1133

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

ponyisi
Copy link
Collaborator

@ponyisi ponyisi commented Aug 2, 2025

Add values.yaml options to allow the specification of a "Virtual Placement Service" site which gives a list of xcache servers to use. In principle this includes also a liveness test, so servers that go down should be removed dynamically from the list we use for transforms. This will be checked if the list hasn't been updated within a certain interval, by default 30 minutes. This new option can fully replace the server list given by cachePrefix.

This is not my favorite way to implement this, but since gunicorn spawns multiple workers which have distinct configuration data, this is the simplest way of ensuring that all app instances will eventually update. It would conceptually be cleaner to store global configuration data that might be updated in a single location (Redis?) but that's a much bigger refactoring.

@ponyisi ponyisi changed the base branch from celery-file-load to develop August 3, 2025 14:14
@ponyisi ponyisi changed the title Xcache-updater Allow list of available xcache servers to be downloaded Aug 3, 2025
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for dynamically retrieving xcache server lists from a Virtual Placement Service (VPS) instead of relying on a static configuration. The implementation periodically checks the VPS liveness service to get an updated list of available servers, with configurable check intervals and automatic fallback handling.

  • Introduces VPS-based xcache server discovery with liveness checking
  • Adds configurable check intervals (default 30 minutes) for server list updates
  • Provides Helm chart configuration options for VPS site specification

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File Description
transformer_manager.py Implements VPS server validation logic and dynamic cache prefix updates
values.yaml Adds configuration options for VPS site, check interval, and liveness URL
configmap.yaml Maps VPS configuration values to environment variables

Comment on lines +419 to +421
import time
import urllib3

Copy link
Preview

Copilot AI Aug 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imports should be placed at the top of the file rather than inside functions. Move the time and urllib3 imports to the module level.

Suggested change
import time
import urllib3

Copilot uses AI. Check for mistakes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant