Reboot control host before provisioning

[p-gentili]

Description

1. Any type of control host might affect provisioning
2. Any type of control host might affect testing
3. If the control host is not reachable, (1) or (2) can fail for a reason completed unrelated to the DUT

This PR propose a programmatic way to reboot the control host just before provisioning in case it's not reachable.

This is still opt-in (you need to call the super() method), but I think it's safe to call it everywhere for now, since it doesn't raise any error. We could also wrap it in try/except Exception if it makes sense.

Also, connectors relying on a specific control host, now also wait for the service of interest to be available.

Resolved issues

N/A

Documentation

N/A

Web service API changes

No!

Tests

Test with functional control host machine:

2026-01-15 11:27:22,323 rpi5b8g001 INFO: DEVICE CONNECTOR: Running pre-provision hook
2026-01-15 11:27:22,323 rpi5b8g001 INFO: DEVICE CONNECTOR: Waiting for a running SSH server on control host 10.102.246.37
2026-01-15 11:27:22,682 rpi5b8g001 INFO: DEVICE CONNECTOR: BEGIN provision
2026-01-15 11:27:22,682 rpi5b8g001 INFO: DEVICE CONNECTOR: Provisioning device
2026-01-15 11:27:22,684 rpi5b8g001 INFO: DEVICE CONNECTOR: Waiting for a running RPyC server on control host xxx
2026-01-15 11:27:41,658 rpi5b8g001 WARNING: DEVICE CONNECTOR: Device /dev/sda might not be mounted
...

Test with not reachable control host (Cannot run snmp commands remotely, from source) :

2026-01-15 11:33:03,775 rpi5b8g001 INFO: DEVICE CONNECTOR: Running pre-provision hook
2026-01-15 11:33:03,775 rpi5b8g001 INFO: DEVICE CONNECTOR: Waiting for a running SSH server on control host 10.102.246.37
2026-01-15 11:33:06,780 rpi5b8g001 INFO: DEVICE CONNECTOR: Running control host reboot script
2026-01-15 11:33:06,780 rpi5b8g001 INFO: DEVICE CONNECTOR: Executing: snmpset -c private -v 2c XXX 1.3.6.1.2.1.105.1.1.1.3.0.2 i 2
2026-01-15 11:33:06,781 rpi5b8g001 ERROR: DEVICE CONNECTOR: Command failed: snmpset -c private -v 2c XXX 1.3.6.1.2.1.105.1.1.1.3.0.2 i 2 (exit code: 127)
2026-01-15 11:33:06,781 rpi5b8g001 ERROR: DEVICE CONNECTOR: stderr: /bin/sh: 1: snmpset: not found

2026-01-15 11:33:06,781 rpi5b8g001 INFO: DEVICE CONNECTOR: Executing: sleep 5
2026-01-15 11:33:11,784 rpi5b8g001 INFO: DEVICE CONNECTOR: Command completed successfully: sleep 5
2026-01-15 11:33:11,784 rpi5b8g001 INFO: DEVICE CONNECTOR: Executing: snmpset -c private -v 2c XXX 1.3.6.1.2.1.105.1.1.1.3.0.2 i 1
2026-01-15 11:33:11,785 rpi5b8g001 ERROR: DEVICE CONNECTOR: Command failed: snmpset -c private -v 2c XXX 1.3.6.1.2.1.105.1.1.1.3.0.2 i 1 (exit code: 127)
2026-01-15 11:33:11,785 rpi5b8g001 ERROR: DEVICE CONNECTOR: stderr: /bin/sh: 1: snmpset: not found

2026-01-15 11:38:12,230 rpi5b8g001 ERROR: DEVICE CONNECTOR: Control host XXX is not available or the SSH server is not running after 300 seconds
2026-01-15 11:38:12,235 rpi5b8g001 INFO: DEVICE CONNECTOR: BEGIN provision
2026-01-15 11:38:12,235 rpi5b8g001 INFO: DEVICE CONNECTOR: Provisioning device
2026-01-15 11:38:12,236 rpi5b8g001 INFO: DEVICE CONNECTOR: Waiting for a running RPyC server on control host XXX
2026-01-15 11:38:15,688 rpi5b8g001 ERROR: DEVICE CONNECTOR: Error connecting to serial logging server. Retrying in the background...
2026-01-15 11:39:14,480 rpi5b8g001 ERROR: DEVICE CONNECTOR: Cannot reach out the service over RPyC

^ as you can see here, we attempted reboot but failed. Provisioning will continue, because the specific device connector is responsible for checking what it actually needs.

[codecov]

Codecov Report

❌ Patch coverage is 87.56757% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.93%. Comparing base (13da678) to head (cc640cb).
⚠️ Report is 27 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #837      +/-   ##
==========================================
+ Coverage   70.10%   71.93%   +1.82%     
==========================================
  Files         108      108              
  Lines        9477     9859     +382     
  Branches      843      866      +23     
==========================================
+ Hits         6644     7092     +448     
+ Misses       2673     2590      -83     
- Partials      160      177      +17     

Flag	Coverage Δ		*Carryforward flag
agent	72.85% <ø> (ø)		Carriedforward from 15883a1
cli	88.15% <ø> (ø)		Carriedforward from 15883a1
device	56.75% <87.56%> (+5.64%)	⬆️
server	87.71% <ø> (ø)		Carriedforward from 15883a1

*This pull request uses carry forward flags. Click here to find out more.

Components	Coverage Δ
Agent	72.85% <ø> (ø)
CLI	88.15% <ø> (ø)
Common	∅ <ø> (∅)
Device Connectors	56.75% <87.56%> (+5.64%)	⬆️
Server	87.71% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[amalinowski75]

Make sense. Just a few minor comments.

[p-gentili]

Please check again the PR description: I added an additional check in a couple of connectors to wait for RPyC to be available.

[amalinowski75]

Thanks for the changes, looks good.