Module graceful shutdown support #567

rameshraghupathy · 2025-05-14T01:49:12Z

Provide support for SmartSwitch DPU module graceful shutdown.

Description

Single source of truth for transitions
- All components now use sonic_platform_base.module_base.ModuleBase helpers:
  - set_module_state_transition(db, name, transition_type)
  - clear_module_state_transition(db, name)
  - get_module_state_transition(db, name) -> dict
  - is_module_state_transition_timed_out(db, name, timeout_secs) -> bool
- Eliminates duplicated logic and race-prone direct Redis writes.
Correct table everywhere
- Standardized on CHASSIS_MODULE_TABLE (replaces CHASSIS_MODULE_INFO_TABLE).
- HLD mismatch addressed in code (HLD fix tracked separately).
Ownership & lifecycle
- The initiator of an operation (startup/shutdown/reboot) sets:
  - state_transition_in_progress=True
  - transition_type=<op>
  - transition_start_time=<utc-iso8601>
- The platform (set_admin_state()) is responsible for clearing:
  - state_transition_in_progress=False
  - optionally transition_end_time=<epoch> (or similar end stamp).
- CLI pre-clears only when a prior transition is timed out.
Timeouts & policy
- Platform JSON path only: /usr/share/sonic/device/{plat}/platform.json; else constants.
- Typical production values used:
  - startup: 180s, shutdown: 180s (≈ graceful_wait 60s + power 120s), reboot: 120s.
- Graceful wait (e.g., waiting for “Graceful shutdown complete”) is a platform policy and implemented inside platform set_admin_state()—not in ModuleBase.
Boot behavior
- chassisd on start:
  1. Clears stale flags once (centralized sweep).
  2. Runs set_initial_dpu_admin_state() which marks transitions via ModuleBase before calling platform set_admin_state().
  3. Leaves clearing to the platform or to well-defined status transitions (ONLINE/OFFLINE) where appropriate.
gNOI shutdown daemon
- Listens on CHASSIS_MODULE_TABLE and triggers only when:
  - state_transition_in_progress=True and transition_type=shutdown.
- Never clears the flag (ownership stays with the platform).
- Bounded RPC timeouts and robust Redis access (swsssdk/swsscommon).
CLI (config chassis modules …)
- Uses ModuleBase APIs for all set/get/timeout checks.
- If a previous transition is stuck, is_module_state_transition_timed_out() → auto-clear then proceed.
- Sets transition at the start of startup/shutdown; platform clears on completion.
- Fabric card flow retained; edits are surgical.
Redis robustness
- Helpers handle both stacks (swsssdk/swsscommon); no hset(mapping=...) usage.
- Consistent HGETALL/HSET paths; resilient to connector differences.
Race reduction & consistency
- Centralized writes prevent multi-writer races.
- All transition writes include transition_start_time; clears may add an end stamp.
- Existing PCI/file-lock logic left intact; unrelated behavior unchanged.
Change scope
- Minimal, targeted diffs.
- No background tasks added, no broad refactors beyond transition handling.
- Behavior changes are limited to making transition semantics correct and uniform across repos.

HLD: # 1991 sonic-net/SONiC#1991
sonic-host-services: #255 sonic-net/sonic-host-services#255
sonic-platform-daemons: sonic-net/sonic-platform-daemons#667
sonic-utilities: sonic-net/sonic-utilities#4031

How Has This Been Tested?

Issue the "config chassis modules shutdown DPUx" command
Verify the DPU module is gracefully shut by checking the logs in /var/log/syslog on both NPU and DPU

mssonicbld · 2025-05-14T01:49:20Z

/azp run

azure-pipelines · 2025-05-14T01:49:28Z

Azure Pipelines successfully started running 1 pipeline(s).

vvolam · 2025-05-20T18:06:19Z

sonic_platform_base/module_base.py

+    # gnoi reboot pipe related
+    GNOI_REBOOT_PIPE_PATH = "/host/gnoi_reboot.pipe"
+    GNOI_REBOOT_RESPONSE_PIPE_PATH = "/host/gnoi_reboot_response.pipe"
+    GNOI_PORT = 50052


Default port is 8080 and please ready from Redis similar to https://github.com/sonic-net/sonic-utilities/blob/c78e0f73fece3fb1c6fb07718a64eddd337dae23/scripts/reboot_smartswitch_helper#L41C1-L45C2

Not required anymore. Cleaned it.

vvolam · 2025-05-20T18:07:04Z

sonic_platform_base/module_base.py

+    GNOI_REBOOT_PIPE_PATH = "/host/gnoi_reboot.pipe"
+    GNOI_REBOOT_RESPONSE_PIPE_PATH = "/host/gnoi_reboot_response.pipe"
+    GNOI_PORT = 50052
+    GNOI_RESPONSE_TIMEOUT = 60  # seconds


Please read the timeout from platform.json similar to https://github.com/sonic-net/sonic-utilities/blob/c78e0f73fece3fb1c6fb07718a64eddd337dae23/scripts/reboot_smartswitch_helper#L109C7-L109C52

vvolam · 2025-05-20T18:07:39Z

sonic_platform_base/module_base.py

+
+        This method performs the following steps:
+        1. Sends a JSON-formatted reboot request to the gNOI reboot daemon via a named pipe.
+        2. Waits for a response on a designated response pipe, with a timeout of 60 seconds.


update the comment accordingly to platform.json timeout

vvolam · 2025-05-20T18:35:10Z

sonic_platform_base/module_base.py

        """
        raise NotImplementedError

+    def pre_shutdown_hook(self):


Where is this invoked?

refactored and not valid anymore

mssonicbld · 2025-05-21T16:14:04Z

/azp run

azure-pipelines · 2025-05-21T16:14:38Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T17:53:45Z

/azp run

azure-pipelines · 2025-05-21T17:53:54Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T19:31:41Z

/azp run

azure-pipelines · 2025-05-21T19:31:50Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-07-07T17:09:57Z

/azp run

azure-pipelines · 2025-07-07T17:10:03Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

mssonicbld · 2025-07-10T17:28:58Z

/azp run

azure-pipelines · 2025-07-10T17:29:05Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

oleksandrivantsiv · 2025-08-06T21:28:34Z

sonic_platform_base/module_base.py

+            subtype = device_info.get_device_subtype()
+            if subtype == "SmartSwitch" and not is_dpu():
+                self.graceful_shutdown_handler()
+        # Proceed to set the admin state using the platform-specific implementation


How is this supposed to work? super here will call set_admin_state of the base class, not of the derived one.

In the refactored implementation the platform will graceful_shutdown_handler()

mssonicbld · 2025-08-12T18:13:50Z

/azp run

azure-pipelines · 2025-08-12T18:13:59Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-08-12T18:31:06Z

/azp run

azure-pipelines · 2025-08-12T18:31:16Z

Azure Pipelines successfully started running 1 pipeline(s).

Co-authored-by: Copilot <[email protected]>

mssonicbld · 2025-10-21T21:02:13Z

/azp run

azure-pipelines · 2025-10-21T21:02:22Z

Azure Pipelines successfully started running 1 pipeline(s).

Fixing test failures

mssonicbld · 2025-10-21T22:01:23Z

/azp run

azure-pipelines · 2025-10-21T22:01:33Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-23T17:54:27Z

/azp run

azure-pipelines · 2025-10-23T17:54:36Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-23T18:05:28Z

/azp run

azure-pipelines · 2025-10-23T18:05:38Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

sonic_platform_base/module_base.py

Copilot · 2025-10-23T18:19:34Z

tests/module_base_test.py

+        mb.ModuleBase._TRANSITION_TIMEOUTS_CACHE = None
+        with patch("os.path.exists", return_value=False):
+            d = Dummy()
+            assert d._load_transition_timeouts()["reboot"] == 240


Magic number 240 should use the constant from ModuleBase._TRANSITION_TIMEOUT_DEFAULTS["reboot"] to ensure test stays in sync with actual default values and avoid hardcoding the same value in multiple places.

sonic_platform_base/module_base.py

Co-authored-by: Copilot <[email protected]>

mssonicbld · 2025-10-23T18:36:26Z

/azp run

mssonicbld · 2025-10-23T18:36:27Z

/azp run

azure-pipelines · 2025-10-23T18:36:35Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2025-10-23T18:36:36Z

Azure Pipelines successfully started running 1 pipeline(s).

Co-authored-by: Copilot <[email protected]>

mssonicbld · 2025-10-23T18:42:09Z

/azp run

azure-pipelines · 2025-10-23T18:42:19Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-23T18:44:37Z

/azp run

azure-pipelines · 2025-10-23T18:44:46Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-23T18:56:09Z

/azp run

azure-pipelines · 2025-10-23T18:56:18Z

Azure Pipelines successfully started running 1 pipeline(s).

vvolam · 2025-10-25T02:06:50Z

sonic_platform_base/module_base.py

+            return admin_state_success
+
+        # Admin DOWN: Perform graceful shutdown first
+        module_name = self.get_name()


For uniformity, invoke set_module_state_transition here itself before calling graceful shutdown handler

vvolam · 2025-10-25T02:11:10Z

sonic_platform_base/module_base.py

-            PCIE_DETACH_INFO_TABLE_KEY = PCIE_DETACH_INFO_TABLE+"|"+pcie_string
-            if not self.state_db_connector:
-                self.state_db_connector = swsscommon.swsscommon.DBConnector("STATE_DB", 0)
+            db = self._state_db_connector


Just delete line 345 and 346 and simply replace state_db_connector with _state_db_connector or better just use prior name of state_db_connector(), so unnecessary code changes can be avoided.

vvolam · 2025-10-25T02:13:25Z

sonic_platform_base/module_base.py

-            self.state_db_connector.hset(PCIE_DETACH_INFO_TABLE_KEY, "bus_info", pcie_string)
-            self.state_db_connector.hset(PCIE_DETACH_INFO_TABLE_KEY, "dpu_state", operation)
+            # Set the PCI detach info for detaching operation
+            db.set(db.STATE_DB, PCIE_DETACH_INFO_TABLE_KEY, {


hset is the right way to set the keys in the STATE_DB, please avoid unrelated changes.

vvolam · 2025-10-25T02:15:18Z

sonic_platform_base/module_base.py

-                self.state_db_connector.delete(PCIE_DETACH_INFO_TABLE_KEY)
+                # Delete the entire entry for attaching operation
+                if hasattr(db, 'delete'):
+                    db.delete(db.STATE_DB, PCIE_DETACH_INFO_TABLE_KEY, "bus_info")


Line 351 is correct, as the delete deletes entire PCIE_DETACH_INFO_TABLE_KEY

vvolam · 2025-10-25T02:18:14Z

sonic_platform_base/module_base.py

+        # Atomically set transition state (handles race conditions with locking)
+        # Note: This is safe to call even if caller already set transition state,
+        # as the function is idempotent and will not overwrite existing valid transitions
+        self.set_module_state_transition(db, module_name, "shutdown")


As mentioned above, this should be set before calling the graceful handler as we are clearing the flag outside itself

vvolam · 2025-10-25T02:20:18Z

sonic_platform_base/module_base.py

+            try:
+                oper = self.get_oper_status()
+                if oper and str(oper).lower() == "offline":
+                    if not self.clear_module_state_transition(db, module_name):


This seems to be duplicate code as the caller is clearing in False and True case, could you please re-check?

vvolam · 2025-10-25T02:20:57Z

sonic_platform_base/module_base.py

+            # This handles cases where multiple agents might be waiting
+            if self.is_module_state_transition_timed_out(db, module_name, shutdown_timeout):
+                # Clear only if we can confirm it's actually timed out
+                if not self.clear_module_state_transition(db, module_name):


Same here, caller is also clearing the state upon returning, this seems to be duplicate step

vvolam · 2025-10-25T02:21:39Z

sonic_platform_base/module_base.py

+
+        # Final timeout check before clearing - use recorded start time, not our local wait time
+        if self.is_module_state_transition_timed_out(db, module_name, shutdown_timeout):
+            if not self.clear_module_state_transition(db, module_name):


same here, seems to be duplicate

vvolam · 2025-10-25T02:27:24Z

sonic_platform_base/module_base.py

+        if up:
+            # Admin UP: Set transition state to 'startup' before admin state change
+            module_name = self.get_name()
+            self.set_module_state_transition(self._state_db_connector, module_name, "startup")


Should we be aborting the operation if the set fails according to https://github.com/sonic-net/SONiC/blob/master/doc/smart-switch/graceful-shutdown/graceful-shutdown.md HLD "Scenario 1"?

Also, we need to set this transition state before pre-shutdown, because if setting state fails, we need to abort the operation. Should we leave set/clear transitions to the caller to correctly do pre-shutdown and post-startup sequence?

vvolam · 2025-10-25T02:32:25Z

sonic_platform_base/module_base.py

+        if t0.tzinfo is None:
+            t0 = t0.replace(tzinfo=timezone.utc)
+
+        age = (datetime.now(timezone.utc) - t0).total_seconds()


@rameshraghupathy could you address comment from Qi

Added pre_shutdown_hook() function to module_base.py

0409850

rameshraghupathy mentioned this pull request May 14, 2025

Module graceful shutdown support sonic-net/sonic-buildimage#22611

Draft

vvolam reviewed May 20, 2025

View reviewed changes

rameshraghupathy mentioned this pull request May 21, 2025

HLD for SmsrtSwitch DPU graceful shutdown sonic-net/SONiC#1991

Merged

Modified based on the Redis based IPC

69fc737

Did some cleanup

52fed94

Modified set_admin_state API to handle DPU Graceful Shutdown

13ccb54

rameshraghupathy mentioned this pull request Jul 7, 2025

Module graceful shutdown support sonic-net/sonic-host-services#255

Open

Draft version. Need to test again

f80d453

Merge branch 'master' into graceful-shutdown

075f9fe

oleksandrivantsiv suggested changes Aug 6, 2025

View reviewed changes

Merge branch 'sonic-net:master' into graceful-shutdown

5c76c4f

refactored based on the revised HLD

954d205

Update tests/module_base_test.py

2f8e72d

Co-authored-by: Copilot <[email protected]>

t rebase --abort

ee58019

Fixing test failures

Addressing review comments

3e1dd0a

Fixing test failures

61a091b

vvolam requested a review from Copilot October 23, 2025 18:18

Copilot AI reviewed Oct 23, 2025

View reviewed changes

rameshraghupathy and others added 2 commits October 23, 2025 11:34

Update sonic_platform_base/module_base.py

3c09b88

Co-authored-by: Copilot <[email protected]>

Update sonic_platform_base/module_base.py

da94d73

Co-authored-by: Copilot <[email protected]>

Update sonic_platform_base/module_base.py

41f05ce

Co-authored-by: Copilot <[email protected]>

Merge branch 'sonic-net:master' into graceful-shutdown

6e0e7cb

Addressing review comments

ab9680e

vvolam suggested changes Oct 25, 2025

View reviewed changes

Module graceful shutdown support #567

Are you sure you want to change the base?

Module graceful shutdown support #567

Conversation

rameshraghupathy commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How Has This Been Tested?

Uh oh!

mssonicbld commented May 14, 2025

Uh oh!

azure-pipelines bot commented May 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented Jul 7, 2025

Uh oh!

azure-pipelines bot commented Jul 7, 2025

Uh oh!

mssonicbld commented Jul 10, 2025

Uh oh!

azure-pipelines bot commented Jul 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Aug 12, 2025

Uh oh!

azure-pipelines bot commented Aug 12, 2025

Uh oh!

mssonicbld commented Aug 12, 2025

Uh oh!

azure-pipelines bot commented Aug 12, 2025

Uh oh!

mssonicbld commented Oct 21, 2025

Uh oh!

azure-pipelines bot commented Oct 21, 2025

Uh oh!

mssonicbld commented Oct 21, 2025

Uh oh!

azure-pipelines bot commented Oct 21, 2025

Uh oh!

mssonicbld commented Oct 23, 2025

Uh oh!

azure-pipelines bot commented Oct 23, 2025

Uh oh!

mssonicbld commented Oct 23, 2025

Uh oh!

azure-pipelines bot commented Oct 23, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

rameshraghupathy commented May 14, 2025 •

edited

Loading