Module graceful shutdown support #255

rameshraghupathy · 2025-05-14T01:56:43Z

Provide support for SmartSwitch DPU module graceful shutdown.

Description:

Single source of truth for transitions
- All components now use sonic_platform_base.module_base.ModuleBase helpers:
  - set_module_state_transition(db, name, transition_type)
  - clear_module_state_transition(db, name)
  - get_module_state_transition(db, name) -> dict
  - is_module_state_transition_timed_out(db, name, timeout_secs) -> bool
- Eliminates duplicated logic and race-prone direct Redis writes.
Correct table everywhere
- Standardized on CHASSIS_MODULE_TABLE (replaces CHASSIS_MODULE_INFO_TABLE).
- HLD mismatch addressed in code (HLD fix tracked separately).
Ownership & lifecycle
- The initiator of an operation (startup/shutdown/reboot) sets:
  - state_transition_in_progress=True
  - transition_type=<op>
  - transition_start_time=<utc-iso8601>
- The platform (set_admin_state()) is responsible for clearing:
  - state_transition_in_progress=False
  - optionally transition_end_time=<epoch> (or similar end stamp).
- CLI pre-clears only when a prior transition is timed out.
Timeouts & policy
- Platform JSON path only: /usr/share/sonic/device/{plat}/platform.json; else constants.
- Typical production values used:
  - startup: 180s, shutdown: 180s (≈ graceful_wait 60s + power 120s), reboot: 120s.
- Graceful wait (e.g., waiting for “Graceful shutdown complete”) is a platform policy and implemented inside platform set_admin_state()—not in ModuleBase.
Boot behavior
- chassisd on start:
  1. Clears stale flags once (centralized sweep).
  2. Runs set_initial_dpu_admin_state() which marks transitions via ModuleBase before calling platform set_admin_state().
  3. Leaves clearing to the platform or to well-defined status transitions (ONLINE/OFFLINE) where appropriate.
gNOI shutdown daemon
- Listens on CHASSIS_MODULE_TABLE and triggers only when:
  - state_transition_in_progress=True and transition_type=shutdown.
- Never clears the flag (ownership stays with the platform).
- Bounded RPC timeouts and robust Redis access (swsssdk/swsscommon).
CLI (config chassis modules …)
- Uses ModuleBase APIs for all set/get/timeout checks.
- If a previous transition is stuck, is_module_state_transition_timed_out() → auto-clear then proceed.
- Sets transition at the start of startup/shutdown; platform clears on completion.
- Fabric card flow retained; edits are surgical.
Redis robustness
- Helpers handle both stacks (swsssdk/swsscommon); no hset(mapping=...) usage.
- Consistent HGETALL/HSET paths; resilient to connector differences.
Race reduction & consistency
- Centralized writes prevent multi-writer races.
- All transition writes include transition_start_time; clears may add an end stamp.
- Existing PCI/file-lock logic left intact; unrelated behavior unchanged.
Change scope
- Minimal, targeted diffs.
- No background tasks added, no broad refactors beyond transition handling.
- Behavior changes are limited to making transition semantics correct and uniform across repos.

HLD: # 1991 sonic-net/SONiC#1991
sonic-platform-common: #567 sonic-net/sonic-platform-common#567
sonic-utilities: sonic-net/sonic-utilities#4031
sonic-platform-daemons: sonic-net/sonic-platform-daemons#667

How to verify it
Issue the "config chassis modules shutdown DPUx" command
Verify the DPU module is gracefully shut by checking the logs in /var/log/syslog on both NPU and DPU

mssonicbld · 2025-05-14T01:56:51Z

/azp run

azure-pipelines · 2025-05-14T01:57:00Z

Azure Pipelines successfully started running 1 pipeline(s).

hdwhdw · 2025-05-19T20:36:20Z

Do you mind pasting the steps and output for testing (commands) in the PR description

data/debian/sonic-host-services-data.gnoi-reboot.service

scripts/gnoi-reboot-daemon

vvolam · 2025-05-20T18:26:07Z

scripts/gnoi-reboot-daemon

+            try:
+                msg = json.loads(line)
+                dpu_ip = msg["dpu_ip"]
+                port = msg.get("port", 50052)


port information?

@vvolam No longer exists

vvolam · 2025-05-20T18:26:38Z

scripts/gnoi-reboot-daemon

+                msg = json.loads(line)
+                dpu_ip = msg["dpu_ip"]
+                port = msg.get("port", 50052)
+                method = msg.get("method", 1)


HALT method is 3

@vvolam No longer this is valid and we use HALT method 3

mssonicbld · 2025-05-21T16:34:52Z

/azp run

azure-pipelines · 2025-05-21T16:35:01Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T17:28:44Z

/azp run

azure-pipelines · 2025-05-21T17:28:53Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T17:34:20Z

/azp run

azure-pipelines · 2025-05-21T17:34:30Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T17:46:58Z

/azp run

azure-pipelines · 2025-05-21T17:47:07Z

Azure Pipelines successfully started running 1 pipeline(s).

vvolam · 2025-05-22T19:58:02Z

scripts/check_platform.sh

+if [[ "$subtype" == "SmartSwitch" && "$is_dpu" != "True" ]]; then
+    exit 0
+else
+    echo "gnoi-reboot-daemon is intended for SmartSwitch platforms only."


This can be debug level log, we don't want to see this on other platforms unnecessarily.

@vvolam Not valid anymore

scripts/gnoi-reboot-daemon

mssonicbld · 2025-07-07T17:09:26Z

/azp run

azure-pipelines · 2025-07-07T17:09:34Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-07-10T17:40:18Z

/azp run

azure-pipelines · 2025-07-10T17:40:27Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-21T00:44:04Z

/azp run

azure-pipelines · 2025-10-21T00:44:13Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-21T01:34:47Z

/azp run

azure-pipelines · 2025-10-21T01:34:56Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-21T02:32:35Z

/azp run

azure-pipelines · 2025-10-21T02:32:44Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-21T03:26:35Z

/azp run

azure-pipelines · 2025-10-21T03:26:44Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-21T03:47:18Z

/azp run

azure-pipelines · 2025-10-21T03:47:26Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-21T04:32:22Z

/azp run

azure-pipelines · 2025-10-21T04:32:31Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-21T04:49:30Z

/azp run

azure-pipelines · 2025-10-21T04:49:38Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull Request Overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

scripts/wait-for-sonic-core.sh

hdwhdw

Thanks for addressing! LGTM.

Co-authored-by: Copilot <[email protected]>

mssonicbld · 2025-10-21T17:53:05Z

/azp run

Co-authored-by: Copilot <[email protected]>

azure-pipelines · 2025-10-21T17:53:12Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

mssonicbld · 2025-10-21T17:53:15Z

/azp run

azure-pipelines · 2025-10-21T17:53:25Z

Azure Pipelines successfully started running 1 pipeline(s).

Did the instrumentation for gnoi-reboot.service

4e3a096

This was referenced May 14, 2025

Module graceful shutdown support sonic-net/sonic-platform-common#567

Open

Module graceful shutdown support sonic-net/sonic-buildimage#22611

Draft

hdwhdw requested review from hdwhdw and vvolam May 19, 2025 20:33

vvolam reviewed May 20, 2025

View reviewed changes

rameshraghupathy mentioned this pull request May 21, 2025

HLD for SmsrtSwitch DPU graceful shutdown sonic-net/SONiC#1991

Merged

Modified based on the Redis based IPC

4a7e6bf

Modified based on the Redis based IPC

c2f9cb8

made check_platform.sh executable

db7848f

Did some cleanup

f946e72

vvolam reviewed May 22, 2025

View reviewed changes

hdwhdw reviewed May 23, 2025

View reviewed changes

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

Draft version. Need to test again

4434463

Fixing test failure

91897ed

Addressing review comments

8d829cc

rameshraghupathy force-pushed the graceful-shutdown branch from 46fc748 to 8d829cc Compare October 21, 2025 01:34

Addressing review comments

942874c

Addressing review comments

d1533a8

Addressing review comments

8454a37

Addressing review comments

7e3bf57

Addressing review comments

3c93891

vvolam requested a review from Copilot October 21, 2025 16:12

Copilot AI reviewed Oct 21, 2025

View reviewed changes

scripts/wait-for-sonic-core.sh Outdated Show resolved Hide resolved

scripts/wait-for-sonic-core.sh Outdated Show resolved Hide resolved

hdwhdw approved these changes Oct 21, 2025

View reviewed changes

Update scripts/wait-for-sonic-core.sh

b1f6139

Co-authored-by: Copilot <[email protected]>

Update scripts/wait-for-sonic-core.sh

6a76f95

Co-authored-by: Copilot <[email protected]>

Module graceful shutdown support #255

Are you sure you want to change the base?

Module graceful shutdown support #255

Uh oh!

Conversation

rameshraghupathy commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Uh oh!

mssonicbld commented May 14, 2025

Uh oh!

azure-pipelines bot commented May 14, 2025

Uh oh!

hdwhdw commented May 19, 2025

Uh oh!

Uh oh!

Uh oh!

vvolam May 20, 2025

Choose a reason for hiding this comment

Uh oh!

rameshraghupathy Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

vvolam May 20, 2025

Choose a reason for hiding this comment

Uh oh!

rameshraghupathy Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

vvolam May 22, 2025

Choose a reason for hiding this comment

Uh oh!

rameshraghupathy Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Jul 7, 2025

Uh oh!

azure-pipelines bot commented Jul 7, 2025

Uh oh!

mssonicbld commented Jul 10, 2025

Uh oh!

azure-pipelines bot commented Jul 10, 2025

Uh oh!

mssonicbld commented Oct 21, 2025

Uh oh!

azure-pipelines bot commented Oct 21, 2025

Uh oh!

mssonicbld commented Oct 21, 2025

Uh oh!

azure-pipelines bot commented Oct 21, 2025

Uh oh!

mssonicbld commented Oct 21, 2025

Uh oh!

azure-pipelines bot commented Oct 21, 2025

Uh oh!

mssonicbld commented Oct 21, 2025

Uh oh!

azure-pipelines bot commented Oct 21, 2025

Uh oh!

mssonicbld commented Oct 21, 2025

Uh oh!

rameshraghupathy commented May 14, 2025 •

edited

Loading