[UX] Deduplicate config warning messages

[atoniolo76]

Fixes #7874

The warning about misaligned client/server config was appearing twice per command execution. This adds a per-run_id cache to deduplicate warnings within a single command while still showing them for new commands.

Tested:

• Ran 'sky show-gpus --infra k8s' successfully
• Warning now appears once instead of twice
• Later commands with a different run_id still show the warning

• Created _WARNED_DISALLOWED_KEYS_CACHE and lock to keep track of warnings logged for a run_id
• Changed override_skypilot_config() function so that before logging a warning, it checks a little cache keyed by the current command’s run_id
• Resets when run_id changes

Tested (run the relevant ones):

• [*] Code formatting: install pre-commit (auto-check on commit) or bash format.sh
• [*] Any manual or new tests for this PR (please specify below)
  sky show-gpus --infra k8s
• All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
• Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
• Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

[Michaelvll]

Thanks @atoniolo76! Could you help fix the CI failure and add a unit test for your implementation?

[atoniolo76]

@Michaelvll Good to go!

[SeungjinYang]

Thanks @atoniolo76! Couple of minor comments, and the PR should be ready to go.

[SeungjinYang]

What do you think about using common_utils.get_usage_run_id here instead?

[atoniolo76]

Env var is always set but I agree this is cleaner

[atoniolo76]

@SeungjinYang I have implemented your suggestions and updated the associated unit test. Ready to merge?

[SeungjinYang]

@SeungjinYang I have implemented your suggestions and updated the associated unit test. Ready to merge?

There's something that I'm hesitant about with merging this PR. Cache in this case is in-memory, so its state are not shared between processes. In cases such as sky exec as referenced in the issue, the reason the warning message prints twice is because the CLI submits two API requests to the server (one to exec, another to tail the execution log), and there is no guarantee that the two request lands on the same server process. If the two requests don't land on the same server process, then in the perspective of each server process, it would be encountering the run id for the first time - and still print two warning messages.

The part that perplexed me was the fact that the unit test clearly works, and your local test clearly works. Given the explanation above, it makes sense why unit test introduced may not be testing the case where two API request lands on two different processes. As for local test, I believe local API uses a single server process by default, so a locally spun up API server would not run into the issue described above.

A remotely deployed team API server, on the other hand, can and do have multiple server processes, and I think the change proposed may not function reliably in such cases.

I believe to guard against this scenario, the cache would have to be set up in a way that multiple processes can share the cache state.

[atoniolo76]

@SeungjinYang Didn't think about the remote scenario, moved to a shared solution. Made a new pr too, closing this one