RFC: Transport-Agnostic Networking and Source Identification for SensESP

[dirkwa]

RFC: Transport-Agnostic Networking and Source Identification for SensESP

Version: v1
Status: Draft — seeking feedback before implementation
Related issues: #334, #738, #680, #813
Related Signal K server: signalk-server#2162, signalk-server#2379

---

Summary

This RFC proposes two related improvements to SensESP:

1. Source identification — SensESP devices currently identify
   themselves to the Signal K server with only a hostname. This makes
   it hard to distinguish multiple devices or build stable source
   priority configurations. This RFC adds a persistent hardware
   identifier (MAC address) and device type to delta messages and
   access requests, directly addressing the gap @tkurki identified in
   signalk-server#2162.

2. Transport-agnostic networking — lift the networking layer from
   its WiFi-specific implementation to the unified Network API in
   Arduino ESP32 3.x. This makes SensESP transport-agnostic, enabling
   wired Ethernet, WiFi, and future transports (PPP/cellular) through
   a single, cleaner abstraction.

Source identification is a small, self-contained change (~10 lines)
that can land first. The network abstraction builds on it and is
additive — existing WiFi sketches compile and behave identically.

---

Motivation

The community has been asking for this

In issue #334 (December
2020), @joelkoz proposed modularizing the communications layer. @mairas
responded with a clear vision:

"WifiConnectionState would be NetworkConnectionState and would be
applied to wired ethernet, CAN bus, NMEA 0183 (RS-422) and RS-485
as well."

In discussion #680,
users asked for Bluetooth and serial connections as alternatives to WiFi.
The workaround today is to use SensESPMinimalApp and manually wire the
delta queue — it works, but it is not ergonomic.

In issue #813, users
requested deep sleep support — which is much simpler when the networking
layer is modular and WiFi is not assumed to be always-on.

Signal K server needs better source identification from SensESP

In signalk-server#2162,
@tkurki identified a concrete problem with SensESP devices:

"One particular problem is (SensESP) sensors sending data over WebSocket,
where we need better UI to identify sensors. This also ties to
authentication: it is not the ws connection nor ip address that we can
rely on."

PR #2379 (source
priority UX rework) explicitly lists "SensESP / WebSocket sensor
identification" as a follow-up item.

Today, signalk_delta_queue.cpp sends exactly one field to identify
the device:

source["label"] = SensESPBaseApp::get_hostname();

That is the only identification. When a user has multiple SensESP
devices reporting to the same server, the server has no reliable way to
tell them apart or to build a stable source priority configuration
around them.

For comparison, N2K devices include a CAN Name — a 64-bit identifier
with manufacturer, device class, and unique serial. It would be great
if SensESP could provide a similar level of identification over
WebSocket.

The good news is that this fits naturally into the network abstraction
work proposed here. The NetworkInterface API already exposes a stable
MAC address that can serve as a persistent device identifier regardless
of transport.

The platform shift already happened

Issue #738 tracks
migration to ESP-IDF 5.x / Arduino 3.x via pioarduino. PR #801 merged pioarduino support, and
pioarduino_esp32 is now the default build environment. This means
SensESP already runs on Arduino ESP32 3.x, which provides:

• NetworkInterface — a common base class for WiFi STA, WiFi AP,
  Ethernet, and PPP, with methods like connected(), hasIP(),
  localIP(), macAddress(), setHostname()
• Network.onEvent() — a unified event system covering all interface
  types (ARDUINO_EVENT_ETH_GOT_IP, ARDUINO_EVENT_WIFI_STA_GOT_IP, etc.)
• Network.isOnline() — a single call that checks whether any
  interface has a routable IP address
• NetworkClient / NetworkServer / NetworkUDP — transport-agnostic
  socket classes that work identically over WiFi, Ethernet, or PPP

SensESP's higher-level components (esp_websocket_client,
esp_http_client, mDNS, OTA) already operate at the TCP/IP layer and
are transport-agnostic. The only coupling is in the networking setup and
a handful of WiFi.* status calls.

Real hardware exists and is waiting

There is a growing family of ESP32 boards with Ethernet and PoE
designed for exactly this kind of embedded IoT use:

Board	Ethernet PHY	PoE	Notes
Aptinex IsolPoE ESP32	LAN8720A	Isolated 12W	USB-C, 15 GPIOs
Olimex ESP32-POE-ISO	LAN8710A	IEEE 802.3af	Open hardware, industrial
wESP32	LAN8720A	IEEE 802.3at 13W	Purpose-built for wired IoT
Olimex ESP32-GATEWAY	LAN8710A	No	Low cost, Ethernet + WiFi

These boards are well-suited for marine environments where wired
connections are more reliable than WiFi, and PoE eliminates the need for
separate 12V power wiring.

---

What changes

The core insight

SensESP currently touches WiFi.* in only 3 source files and uses
the result in 7 files total. The WebSocket client, HTTP server, mDNS,
and OTA all operate on the TCP/IP stack and do not care about the
underlying transport. The change surface is small.

Proposed architecture

Source identification (addresses signalk-server#2162)

Enrich the delta source object in SKDeltaQueue::get_delta() and
the access request in SKWSClient::send_access_request():

// Delta source object — before
source["label"] = SensESPBaseApp::get_hostname();

// Delta source object — after
source["label"] = SensESPBaseApp::get_hostname();
source["type"] = "SensESP";
source["src"] = WiFi.macAddress();  // stable hardware ID

// Access request — before
doc["description"] = "SensESP device: " + hostname;

// Access request — after
doc["description"] = "SensESP device: " + hostname;
doc["deviceId"] = WiFi.macAddress();
doc["firmwareVersion"] = SENSESP_VERSION;

The MAC address provides a stable identifier that does not change
across reboots, firmware updates, or hostname changes — comparable to
the CAN Name that N2K devices use. The server can use source.src as
a persistent key for source priority configuration.

Initially this uses WiFi.macAddress() directly. Once the network
abstraction is in place, it switches to active_interface->macAddress()
so it works across WiFi, Ethernet, and future transports.

The list of SK paths a device emits is already available via the
existing metadata mechanism (SKEmitter::get_sources() /
add_metadata()). No changes needed there.

This is ~10 lines of new code, but it directly addresses what @tkurki
identified as a gap and makes SensESP devices proper participants in
the source priority system alongside N2K and NMEA 0183 sources.

Network state (replaces WiFiState / WiFiStateProducer)

A new NetworkStateProducer uses Network.onEvent() to listen for
*_GOT_IP and *_DISCONNECTED events from any interface. It emits a
generalized NetworkState:

enum class NetworkState {
  kDisconnected,       // no interface has an IP
  kConnected,          // at least one interface has a routable IP
  kWifiAPModeActivated // WiFi AP active (provisioning mode)
};

// Backward compatibility
using WiFiState = NetworkState;

This replaces WiFiStateProducer which currently listens only to
ARDUINO_EVENT_WIFI_* events. The new producer responds to Ethernet,
WiFi, and PPP events uniformly.

Interface provisioning (split from Networking)

The current Networking class handles five concerns simultaneously:
WiFi STA connection, WiFi AP provisioning, WiFi scanning, captive portal
DNS, and configuration persistence. This RFC proposes splitting it:

• WiFiProvisioner — all existing WiFi functionality: STA connection,
  AP mode, captive portal, scanning, credential storage. No behavior
  change for existing users.
• EthernetProvisioner — calls ETH.begin() with user-supplied pin
  configuration. Supports DHCP (default) and static IP. Approximately
  50 lines of code.

Both provisioners emit their state into the shared NetworkStateProducer.
They are independent modules — you can use one, both, or neither.

Signal K connection (clean up SKWSClient)

The single WiFi gate in signalk_ws_client.cpp line 543:

// Before
if (!WiFi.isConnected() && WiFi.getMode() != WIFI_MODE_AP) {

// After
if (!Network.isOnline()) {

No other changes needed. esp_websocket_client and esp_http_client
are already transport-agnostic.

Status and diagnostics

Status page items and system info sensors switch from WiFi.localIP() /
WiFi.RSSI() / WiFi.macAddress() to the interface-agnostic equivalents
provided by NetworkInterface:

interface->localIP()     // works for WiFi and Ethernet
interface->macAddress()  // works for WiFi and Ethernet
interface->connected()   // works for WiFi and Ethernet

RSSI is only shown when WiFi is the active interface. Ethernet mode shows
link status instead.

Builder API

The SensESPAppBuilder gains new methods alongside the existing ones:

// Existing (unchanged)
builder.set_wifi_client("SSID", "password");
builder.set_wifi_access_point("SensESP", "configure");

// New
builder.set_ethernet(ETH_PHY_LAN8720, 0, 23, 18, 17, ETH_CLOCK_GPIO17_OUT);

// Dual mode: WiFi AP for provisioning + Ethernet for data
builder.set_wifi_access_point("SensESP", "configure");
builder.set_ethernet(ETH_PHY_LAN8720, 0, 23, 18, 17, ETH_CLOCK_GPIO17_OUT);

Files affected

File	Change	Nature
net/wifi_state.h	Rename to network_state.h, generalize enum, add backward compat alias	Rename
net/networking.h/.cpp	Rename to wifi_provisioner.h/.cpp, extract NetworkStateProducer	Split
net/network_state_producer.h	New — ~40 lines, listens to Network.onEvent()	New
net/ethernet_provisioner.h/.cpp	New — ~50 lines, ETH.begin() + config	New
controllers/system_status_controller.h	Accept NetworkState instead of WiFiState	Trivial
signalk/signalk_ws_client.cpp	Line 543: Network.isOnline(); enrich access request with deviceId	Minor
signalk/signalk_delta_queue.cpp	Add type and src (MAC) to delta source object	Minor
net/http_server.h/.cpp	Remove #include "WiFi.h", captive portal only when WiFi AP active	Minor
sensors/system_info.h/.cpp	Use NetworkInterface* for IP/MAC	Minor
sensesp_app.h	Hold NetworkStateProducer, optional provisioner(s), adapt status items	Moderate
sensesp_app_builder.h	Add set_ethernet(), keep existing WiFi API unchanged	Additive
net/web/app_command_handler.*	WiFi scan handlers only registered when WiFiProvisioner present	Minor
platformio.ini	Add env for Ethernet-capable boards, consider dropping legacy [arduino] env	Config

The net code delta is approximately +100 lines of new logic
(NetworkStateProducer + EthernetProvisioner) with simplification
elsewhere. Existing WiFi behavior is preserved — this is additive.

---

What this enables

Immediate benefits

• Reliable multi-device identification — users running multiple
  SensESP devices can distinguish them on the server and configure
  stable source priorities, just like N2K devices.
• Wired Ethernet with PoE — single-cable installations in engine rooms,
  mast bases, lazarettes. More reliable than WiFi in electrically noisy
  marine environments.
• Ethernet + BLE coexistence — with WiFi disabled, BLE gets exclusive
  access to the ESP32's shared RF module. No coexistence issues, better
  range. Ideal for BLE sensor gateways (Victron SmartShunt, Ruuvi tags,
  Xiaomi sensors) feeding data into Signal K.
• Simpler wired installations — no SSID configuration, no WiFi
  password management, no connectivity drops. Plug in Ethernet, get DHCP,
  connect to Signal K.
• WiFi AP for config + Ethernet for data — provision via phone, run
  via wire. Best of both worlds.

Architectural benefits

• Cleaner separation of concerns — provisioning (how you get on the
  network) is separated from connectivity (whether you are on the network)
  is separated from communication (talking to Signal K). Each can evolve
  independently.
• Easier to add future transports — PPP for cellular modems
  (offshore/bluewater boats with satellite or LTE), serial-over-USB, or
  even CAN-based connections. Each is just a new provisioner.
• Simpler deep sleep support (#813)
  — when the networking layer is modular, wake-connect-send-sleep cycles
  are much easier to implement without WiFi assumptions baked into the
  core.
• Fulfills the vision of #334
  — the communications layer modularization that has been on the roadmap
  since 2020.

Community benefits

• Lowers the barrier for PoE board users — a growing segment of ESP32
  hardware that currently cannot use SensESP without custom workarounds.
• Marine-grade reliability — wired Ethernet is standard practice in
  professional marine electronics (NMEA networks are wired for good
  reason). It would be nice if SensESP could offer this too.
• New use cases — BLE-to-SignalK gateways, PoE-powered remote sensor
  nodes, multi-interface installations.

---

What does NOT change

• Existing WiFi user code — set_wifi_client(), set_wifi_access_point(),
  enable_ota(), all system info sensors. Existing sketches compile and
  run without modification.
• Signal K protocol handling — SKWSClient, SKDeltaQueue,
  SKOutput, SKListener. All unchanged except for richer source
  metadata in delta messages (additive, backward compatible).
• Sensor framework — transforms, consumers, producers. All unchanged.
• Web UI — configuration interface continues to work. WiFi scan page
  only appears when WiFi provisioner is active.
• Build system — pioarduino remains the default. Board-specific envs
  are additive.

---

Open questions

1. Drop the legacy [arduino] env (espressif32 @ ^6.9.0)?
   Arduino ESP32 2.x does not have NetworkInterface or the unified
   Network API. Supporting it would require a compatibility shim. Given
   that pioarduino is already the default and the official PlatformIO
   ESP32 platform is stalled, is it time to make Arduino 3.x the minimum
   requirement?

2. Versioning — this is a breaking change for the Networking class
   name and WiFiState enum (though backward-compat aliases are
   proposed). Should this be SensESP 4.0, or can it land in 3.3.0 with
   deprecation notices?

3. Board-specific pin presets — should the library include predefined
   pin configurations for popular PoE boards (Olimex, wESP32, Aptinex)?
   Or leave that to examples?

4. Frontend/web UI — should the status page adapt dynamically based
   on active interface (hide SSID/RSSI when on Ethernet, show link
   speed instead)? Or is a static layout acceptable for now?

5. Source identification fields — the delta source object is
   defined loosely in the Signal K spec. What fields should SensESP
   include beyond label? Proposed: type ("SensESP"), src
   (MAC address as stable ID). Should firmwareVersion go in
   source or only in the access request? This should align with
   whatever @tkurki settles on for the unified source model in
   signalk-server#2162.

---

Implementation plan

The work can be done incrementally in reviewable PRs. I would suggest
starting with source identification — it is small, immediately useful,
and connects to the server-side work already in progress. The network
abstraction then follows as a series of focused refactoring steps.

Phase 1 — Source identification

1. PR 1: Enrich source identification
   Add type and src (MAC) to the delta source object. Add deviceId
   and firmwareVersion to the access request. ~10 lines of new code.
   Addresses signalk-server#2162.
   Can use WiFi.macAddress() initially and switch to
   NetworkInterface once available.

Phase 2 — Network abstraction

2. PR 2: Introduce NetworkState and NetworkStateProducer
   Rename the enum, create the unified event listener, add backward
   compat aliases. Wire into SystemStatusController. No behavior change.

3. PR 3: Extract WiFiProvisioner from Networking
   Move WiFi-specific code (AP, STA, scan, captive portal) into
   WiFiProvisioner. SensESPApp uses it identically. Existing user
   code unchanged.

4. PR 4: Fix the SKWSClient WiFi gate
   Replace WiFi.isConnected() with Network.isOnline(). One line.

5. PR 5: Add EthernetProvisioner and builder support
   New class, new builder method, example sketch for a PoE board.

6. PR 6: Generalize system info and status page
   Use NetworkInterface* for IP/MAC/status reporting. Update PR 1's
   source identification to use the transport-agnostic interface.

Each PR is independently reviewable and testable. PRs 2–4 are pure
refactoring with no behavior change, reducing risk.

---

Willingness to contribute

I am the author of signalk-server PR #2379
(source priority UX rework). While working on that, it became clear that
ESP device identification is an open topic on the server side — @tkurki
mentioned it explicitly. The source identification improvement (PR 1)
would be a natural starting point if you are interested, since it is
small and self-contained.

With source identification landed, the network abstraction (PRs 2–6)
follows naturally. I have an Aptinex IsolPoE ESP32
board (ESP32 + LAN8720A + isolated PoE) on hand for testing Ethernet
support on real hardware.

I would appreciate early feedback on the approach — particularly around
the open questions above — before writing code, so that the
implementation aligns with the project's direction.

This builds on the excellent groundwork already in place: the pioarduino
migration, the SensESPMinimalApp decoupling, the WiFiStateProducer
design that was explicitly made to be replaceable, and the reactive
producer/consumer architecture that makes swapping components clean.

---

References

• Arduino ESP32 Network API documentation
• Arduino ESP32 Ethernet API documentation
• Arduino ESP32 NetworkInterface header
• ESP32 RF coexistence guide (WiFi/BLE shared RF)
• SensESP issue #334 — Modularize protocols and communications layer
• SensESP issue #738 — Migrate to ESP-IDF 5.x and Arduino 3.x
• SensESP discussion #680 — Connect by bluetooth or USB serial
• SensESP issue #813 — Deep sleep support
• SensESP PR #801 — Pioarduino support
• signalk-server issue #2162 — Source priority and identification
• signalk-server PR #2379 — Source priority UX rework (lists SensESP identification as follow-up)

[tkurki]

I think you've misunderstood me. hostname is the persistent, user assigned identity of the SensESP device already, as @JohnySeven pointed out in Discord. The shortcoming has been on the serverside, where the server has not made any use of it.

persistent hardware
identifier (MAC address) and device type to delta messages and
access requests

I am not sure if this is the correct strategy. Alternative would be to have a connection level identity of the device used as $source, derived from authentication or client hello message.

Imho this PR scope is way too wide and part of it is not SensESP specific - ws source identification should be addressed separately in server/specification and only the networking low level details here.

[dirkwa]

Thanks @tkurki — fair points on scope. I agree that ws source identification should be addressed at the server/spec level, and I'll narrow this RFC to just the networking abstraction.

On hostname being the persistent identity: I agree it's user-assigned and survives reboots. But there's a concrete scenario that I think still needs something from the device side: two boards flashed from the same example sketch, security disabled, user hasn't changed the hostname. Both connect as "SensESP", both send source.label = "SensESP". The server can't tell them apart — and can't discover anything to distinguish them either, since even the MAC address isn't visible beyond the local subnet (ARP doesn't cross routers).

A client hello in the WS handshake would be the right mechanism (agreed, that's server/spec), but the device still needs to put something unique in it. The MAC address is the only identifier that's unique per board, available without user action, and available without authentication. That's why I still think including it — not in every delta, but in the hello/handshake — would be worth considering when designing the spec-level solution.

[dirkwa]

@tkurki

Playing with BLE Provider API POC https://discord.com/channels/1170433917761892493/1478948783630057596 it becomes more and more clear that hostname alone is not enough when flashing 3x ESP32 in a row, than forgot one hostname change and start over again, after looking at the switch ports where I placed the duplicate 🙄