Enhancement Request: implement Retry with Backoff pattern for network I/O requests

[klmcwhirter]

Overview

Relying on network-online.target only is problematic as documented in several places.

The uupd.{timer,service} units rely on network-online.target. But that does not mean what it sounds like. It does not mean that the network is ready for traffic. See https://systemd.io/NETWORK_ONLINE/.

I should mention that I use wifi exclusively - not wired ethernet.

When opening the laptop lid (Wake from Sleep), I keep seeing errors like this:

Jan 15 04:57:11 nitro5 uupd[157219]: {"level":"ERROR","msg":"Hardware checks failed","error":"Network, returned error: network not online"}

Hence, the need for the Retry with Backoff pattern to be implemented in the uupd code itself.

Note that the Circuit Breaker pattern might also be beneficial for situations like CloudFlare introducing a change that disallowed connectivity to dependent assets.

Example Use Case

I have written software, that is similarly scheduled by systemd where the solution was to implement the Retry with Backoff pattern to prevent false positive reporting.

The systemd units also rely on network-online.target. And because requirements allowed for it, also sets the BindsTo=gnome-session.target property as I was guided in a forum somewhere; although I was not able to make that advice work for me. I do realize that the uupd requirements do not allow for assuming an authenticated gnome session.

Once I implemented the Retry with Backoff pattern in the rule (written in python) needing to use ssh reliably, I started to see log entries like the following and resilience was improved.

Feb 02 05:24:36 nitro5 upd_monitor.sh[390387]: 2026-02-02 05:24:36,796 - steps - WARNING - process_rule - Worker-04: <class '__mp_main__.ProcessNonZeroRetcodeError'>, Retrying in 3 seconds...
Feb 02 05:24:36 nitro5 upd_monitor.sh[390387]: <class '__mp_main__.ProcessNonZeroRetcodeError'>, Retrying in 6 seconds...
Feb 02 05:24:36 nitro5 upd_monitor.sh[390387]: 2026-02-02 05:24:36,796 - steps - DEBUG - process_rule - Worker-04: Processing pi-cluster-health ... done
Feb 02 05:24:36 nitro5 upd_monitor.sh[390326]: 2026-02-02 05:24:36,796 - steps - INFO - process - Received result from Worker-04
Feb 02 05:24:36 nitro5 upd_monitor.sh[390326]: 2026-02-02 05:24:36,831 - utils - DEBUG - _ - process done
Feb 02 05:24:36 nitro5 systemd[3708]: upd-indicator-monitor.service: Consumed 3.901s CPU time, 656.2M memory peak.

Questions

1. Does this resound with the team's sensibilities regarding industry best practices?
2. Do you agree that the uupd golang code is the correct architectural location for implementing these kinds of reliability / resilience policies?
3. Is the Wake from Sleep event something you want to officially support? As I mentioned elsewhere, using ujust update as a manual workaround has worked fine for me throughout 2025.

Disclaimer

Although I have quite a bit of experience guiding teams to implement reliability and resilience policies in their application architecture on my day job, I will not pretend to claim a similar level of familiarity here. I.e., my experience guides me to know it is needed, but not how to implement it in a Universal Blue system component.

My hope is to encourage discussion; not complain.

Current Version

$ sudo bootc status
● Booted image: ghcr.io/ublue-os/bluefin-dx-nvidia-open:stable
        Digest: sha256:43279a9dfad55057c3f977a33b5d0b64e3e659fb6565c371f449af190dfc20ca (amd64)
       Version: 43.20260127 (2026-01-27T01:52:09Z)

Related Issue

Note that part of the problem was resolved with #126.

[gerblesh]

seems fine to me if it resolves the issue when waking from sleep. To clarify you mean retrying within the uupd hw checks itself right?

[klmcwhirter]

That would probably be the simplest naive approach, yes. But, it would not guarantee reliability. What happens if the H/W checks pass and then the network connection is lost later - intermittently?

Resilience is about more than just asserting required resources are available at the beginning of the process. Self-healing (if that is a requirement) risks can occur in any step.

I lack insight into Universal Blue future vision details and how that would impact the uupd requirements / implementation.

The last time this came up in one of my projects we centralized the policy rules into a separate library that abstracted the team's rule definitions and the underlying 3rd party dependencies so that the resulting library could be reused across the different solutions we maintained. We then built I/O helper functions at public parts of the library to simplify usage for the junior developers on the team.

All I/O used that library - db, filesystem, Kafka / Tibco EMS messaging, etc.

But that seems heavy handed for what is needed here.

With the dakota raptor re-architecture going on ... I'll defer that question to a team member more involved in the bowels, so to speak (but pun intended) of the architecture work for that effort.

It is an important question though.

Let me remind you of these questions as well:

• should Wake on Sleep be officially supported?
• should the definition of network-online.target be enhanced? A new target be added that means network ready for use?
• is there a library in the golang ecosystem that should be used for resilience policy enforcement? (e.g., something like Polly that we used on the day job for .NET solutions)

[gerblesh]

realistically all we need to do is retry the network checks as the update program is run once every day. It would certainly be cool to use a framework or design our own resilience system but that seems pretty overkill for uupd, feel free to leave feedback on the PR here: #139

[klmcwhirter]

realistically all we need to do is retry the network checks as the update program is run once every day. It would certainly be cool to use a framework or design our own resilience system but that seems pretty overkill for uupd, feel free to leave feedback on the PR here: #139

I totally agree with you. To which depth the project desires to apply resiliency patterns is a judgement call based on a lot of factors. And it will more than likely change as the dakota raptor re-architecture effort gets close to completion anyway.

So without any good way to provide useful feedback on the technical implementation details, I'll just say that from a project mgmt perspective I agree with your idea to minimize the scope of the change.

Please feel free to close this ticket when it has lost it's usefulness.