Merge pull request #46 from theelims/eventsocket

Release Event Socket as 0.4.0

Author: theelims

.gitignore

@@ -16,3 +16,4 @@ node_modules
 *WWWData.h
 lib/framework/WWWData.h
 ssl_certs/cacert.pem
+/logs

CHANGELOG.md

@@ -2,37 +2,52 @@
 
 All notable changes to this project will be documented in this file.
 
-## [WIP] - Work in Progress
+## [0.4.0] - 2024-04-21
+
+This upgrade might require one minor change as `MqttPubSub.h` and its class had been renamed to `MqttEndpoint.h` and `MqttEndoint` respectively. However, it is strongly advised, that you change all existing WebSocketServer endpoints to the new event socket system.
+
+> [!NOTE]
+> The new Event Socket system is likely to change with coming updates.
 
 ### Added
 
 - Added build flag `-D SERIAL_INFO` to platformio.ini to enable / disable all `Serial.print()` statements. On some boards with native USB those Serial prints have been reported to block and make the server unresponsive.
-- Added a hook handler to StatefulService. Unlike an UPDATE a hook is called every time a state receives and updated, even if the result is UNCHANGED or ERROR.
-- Added missing include for S2 in SystemStatus.cpp (#23)
-- Added awareness of front end build script for all 3 major JS package managers. The script will auto-identify the package manager by the lock-file. (#40)
+- Added a hook handler to StatefulService. Unlike an UPDATE a hook is called every time a state receives an updated, even if the result is UNCHANGED or ERROR.
+- Added missing include for S2 in SystemStatus.cpp [#23](https://github.com/theelims/ESP32-sveltekit/issues/23)
+- Added awareness of front end build script for all 3 major JS package managers. The script will auto-identify the package manager by the lock-file. [#40](https://github.com/theelims/ESP32-sveltekit/pull/40)
+- Added a new event socket to bundle the websocket server and the notifications events. This saves on open sockets and allows for concurrent visitors of the internal website. The normal websocket server endpoint remains as an option, should a pure websocket connection be desired. An EventEndpoint was added to use this with Stateful Services. [#29](https://github.com/theelims/ESP32-sveltekit/issues/29) and [#43](https://github.com/theelims/ESP32-sveltekit/pull/43)
+- TS Types definition in one central place for the frontend.
 
 ### Changed
 
-- more generic board definition in platformio.ini (#20)
-- refactored MqttPubSub.h into a single class to improve readability
-- Moves appName and copyright to `layout.ts` to keep customization in one place (#31)
-- Make eventSource use timeout for reconnect (#34)
-- Make each toasts disappear after timeout (#35)
+- more generic board definition in platformio.ini [#20](https://github.com/theelims/ESP32-sveltekit/pull/20)
+- Renamed `MqttPubSub.h` and class to `MqttEndpoint.h` and class.
+- refactored MqttEndpoint.h into a single class to improve readability
+- Moves appName and copyright to `layout.ts` to keep customization in one place [#31](https://github.com/theelims/ESP32-sveltekit/pull/31)
+- Make event source use timeout for reconnect [#34](https://github.com/theelims/ESP32-sveltekit/pull/34)
+- Make each toasts disappear after timeout [#35](https://github.com/theelims/ESP32-sveltekit/pull/35)
 - Fixed version `platform = espressif32 @ 6.6.0` in platformio.ini
+- Analytics data limited to 1000 data points (roughly 33 minutes).
+- postcss.config.cjs as ESM module [#24](https://github.com/theelims/ESP32-sveltekit/issues/24)
 
 ### Fixed
 
-- Duplicate method in FeatureService (#18)
 - Fixed compile error with FLAG `-D SERVE_CONFIG_FILES`
-- Fixed typo in telemetry.ts (#38)
-- Fixed the development warning: `Loading /rest/features using `window.fetch`. For best results, use the `fetch`that is passed to your`load` function:`
+- Fixed typo in telemetry.ts [#38](https://github.com/theelims/ESP32-sveltekit/pull/38)
+- Fixed the development warning: `Loading /rest/features using 'window.fetch'. For best results, use the 'fetch' that is passed to your 'load' function:`
 
 ### Removed
 
+- Duplicate method in FeatureService [#18](https://github.com/theelims/ESP32-sveltekit/pull/18)
 - Duplicate lines in Systems Settings view.
-- Removes duplicate begin (#36)
+- Removes duplicate begin [#36](https://github.com/theelims/ESP32-sveltekit/pull/36)
+- Temporary disabled OTA progress update due to crash with PsychicHttp [#32](https://github.com/theelims/ESP32-sveltekit/issues/32) until a fix is found.
+
+### Known Issues
+
+- On ESP32-C3 the security features should be disabled in features.ini: `-D FT_SECURITY=0`. If enabled the ESP32-C3 becomes extremely sluggish with frequent connection drops.
 
-## [0.3.0] - 2023-02-05
+## [0.3.0] - 2024-02-05
 
 > [!CAUTION]
 > This update has breaking changes!

README.md

@@ -23,7 +23,7 @@ SvelteKit is ideally suited to be served from constrained devices like an ESP32.
 
 ### :telephone: Rich Communication Interfaces
 
-Comes with a rich set of communication interfaces to cover most standard needs of an IoT application. Like MQTT client, HTTP RESTful API or WebSocket Server. All communication channels are stateful and fully synchronized. Changes propagate and are communicated to all other participants. The states can be persisted on the file system as well. For accurate time keeping time can by synchronized over NTP.
+Comes with a rich set of communication interfaces to cover most standard needs of an IoT application. Like MQTT client, HTTP RESTful API, a WebSocket based Event Socket and a classic Websocket Server. All communication channels are stateful and fully synchronized. Changes propagate and are communicated to all other participants. The states can be persisted on the file system as well. For accurate time keeping time can by synchronized over NTP.
 
 ### :file_cabinet: WiFi Provisioning and Management
 

docs/buildprocess.md

@@ -141,6 +141,16 @@ build_flags =
   -D SERVE_CONFIG_FILES
 ```
 
+### Serial Info
+
+In some circumstances it might be beneficial to not print any information on the serial consol (Serial1 or USB CDC). By commenting out the following build flag ESP32-Sveltekit will not print any information on the serial console.
+
+```ini
+build_flags =
+...
+  -D SERIAL_INFO
+```
+
 ## SSL Root Certificate Store
 
 Some features like firmware download or the MQTT client require a SSL connection. For that the SSL Root CA certificate must be known to the ESP32. The build system contains a python script derived from Espressif ESP-IDF building a certificate store containing one or more certificates. In order to create the store you must uncomment the three lines below in `platformio.ini`.

docs/index.md

@@ -31,7 +31,7 @@ SvelteKit is ideally suited to be served from constrained devices like an ESP32.
 
 ### :telephone: Rich Communication Interfaces
 
-Comes with a rich set of communication interfaces to cover most standard needs of an IoT application. Like MQTT client, HTTP RESTful API or WebSocket Server. All communication channels are stateful and fully synchronized. Changes propagate and are communicated to all other participants. The states can be persisted on the file system as well. For accurate time keeping time can by synchronized over NTP.
+Comes with a rich set of communication interfaces to cover most standard needs of an IoT application. Like MQTT client, HTTP RESTful API, a WebSocket based Event Socket and a classic Websocket Server. All communication channels are stateful and fully synchronized. Changes propagate and are communicated to all other participants. The states can be persisted on the file system as well. For accurate time keeping time can by synchronized over NTP.
 
 ### :file_cabinet: WiFi Provisioning and Management
 

docs/statefulservice.md

@@ -54,6 +54,8 @@ class LightStateService : public StatefulService<LightState> {
 };
 ```
 
+### Update Handler
+
 You may listen for changes to state by registering an update handler callback. It is possible to remove an update handler later if required.
 
 ```cpp
@@ -74,9 +76,11 @@ An "originId" is passed to the update handler which may be used to identify the
 | Origin                     | Description                                     |
 | -------------------------- | ----------------------------------------------- |
 | http                       | An update sent over REST (HttpEndpoint)         |
-| mqtt                       | An update sent over MQTT (MqttPubSub)           |
+| mqtt                       | An update sent over MQTT (MqttEndpoint)         |
 | websocketserver:{clientId} | An update sent over WebSocket (WebSocketServer) |
 
+### Hook Handler
+
 Sometimes if can be desired to hook into every update of an state, even if the StateUpdateResult is `StateUpdateResult::UNCHANGED` and the update handler isn't called. In such cases you can use the hook handler. Similarly it can be removed later.
 
 ```cpp
@@ -119,7 +123,7 @@ There are three possible return values for an update function which are as follo
 | StateUpdateResult::UNCHANGED | The state was unchanged, propagation should not take place               |
 | StateUpdateResult::ERROR     | There was an error updating the state, propagation should not take place |
 
-### Serialization
+### JSON Serialization
 
 When reading or updating state from an external source (HTTP, WebSockets, or MQTT for example) the state must be marshalled into a serializable form (JSON). SettingsService provides two callback patterns which facilitate this internally:
 
@@ -165,7 +169,7 @@ JsonObject jsonObject = jsonDocument.as<JsonObject>();
 lightStateService->update(jsonObject, LightState::update, "timer");
 ```
 
-### Endpoints
+### HTTP RESTful Endpoint
 
 The framework provides an [HttpEndpoint.h](https://github.com/theelims/ESP32-sveltekit/blob/main/lib/framework/HttpEndpoint.h) class which may be used to register GET and POST handlers to read and update the state over HTTP. You may construct an HttpEndpoint as a part of the StatefulService or separately if you prefer.
 
@@ -191,7 +195,7 @@ Endpoint security is provided by authentication predicates which are [documented
 
 To register the HTTP endpoints with the web server the function `_httpEndpoint.begin()` must be called in the custom StatefulService Class' own `void begin()` function.
 
-### Persistence
+### File System Persistence
 
 [FSPersistence.h](https://github.com/theelims/ESP32-sveltekit/blob/main/lib/framework/FSPersistence.h) allows you to save state to the filesystem. FSPersistence automatically writes changes to the file system when state is updated. This feature can be disabled by calling `disableUpdateHandler()` if manual control of persistence is required.
 
@@ -209,7 +213,33 @@ class LightStateService : public StatefulService<LightState> {
 };
 ```
 
-### WebSockets
+### Event Socket Endpoint
+
+[EventEndpoint.h](https://github.com/theelims/ESP32-sveltekit/blob/main/lib/framework/EventEndpoint.h) wraps the [Event Socket](#event-socket) into an endpoint compatible with a stateful service. The client may subscribe and unsubscribe to this event to receive updates or push updates to the ESP32. The current state is synchronized upon subscription.
+
+The code below demonstrates how to extend the LightStateService class to provide an WebSocket:
+
+```cpp
+class LightStateService : public StatefulService<LightState> {
+ public:
+  LightStateService(EventSocket *socket) :
+      _eventEndpoint(LightState::read, LightState::update, this, socket, "led") {}
+
+  void begin()
+  {
+    _eventEndpoint.begin();
+  }
+
+ private:
+  EventEndpoint<LightState> _eventEndpoint;
+};
+```
+
+To register the event endpoint with the event socket the function `_eventEndpoint.begin()` must be called in the custom StatefulService Class' own `void begin()` function.
+
+Since all events run through one websocket connection it is not possible to use the [securityManager](#security-features) to limit access to individual events. The security defaults to `AuthenticationPredicates::IS_AUTHENTICATED`.
+
+### WebSocket Server
 
 [WebSocketServer.h](https://github.com/theelims/ESP32-sveltekit/blob/main/lib/framework/WebSocketServer.h) allows you to read and update state over a WebSocket connection. WebSocketServer automatically pushes changes to all connected clients when state is updated.
 
@@ -235,11 +265,11 @@ WebSocket security is provided by authentication predicates which are [documente
 
 To register the WS endpoint with the web server the function `_webSocketServer.begin()` must be called in the custom StatefulService Class' own `void begin()` function.
 
-### MQTT
+### MQTT Client
 
 The framework includes an MQTT client which can be configured via the UI. MQTT requirements will differ from project to project so the framework exposes the client for you to use as you see fit. The framework does however provide a utility to interface StatefulService to a pair of pub/sub (state/set) topics. This utility can be used to synchronize state with software such as Home Assistant.
 
-[MqttPubSub.h](https://github.com/theelims/ESP32-sveltekit/blob/main/lib/framework/MqttPubSub.h) allows you to publish and subscribe to synchronize state over a pair of MQTT topics. MqttPubSub automatically pushes changes to the "pub" topic and reads updates from the "sub" topic.
+[MqttEndpoint.h](https://github.com/theelims/ESP32-sveltekit/blob/main/lib/framework/MqttEndpoint.h) allows you to publish and subscribe to synchronize state over a pair of MQTT topics. MqttEndpoint automatically pushes changes to the "pub" topic and reads updates from the "sub" topic.
 
 The code below demonstrates how to extend the LightStateService class to interface with MQTT:
 
@@ -248,7 +278,7 @@ The code below demonstrates how to extend the LightStateService class to interfa
 class LightStateService : public StatefulService<LightState> {
  public:
   LightStateService(AsyncMqttClient* mqttClient) :
-      _mqttPubSub(LightState::read,
+      _mqttEndpoint(LightState::read,
                   LightState::update,
                   this,
                   mqttClient,
@@ -257,18 +287,68 @@ class LightStateService : public StatefulService<LightState> {
   }
 
  private:
-  MqttPubSub<LightState> _mqttPubSub;
+  MqttEndpoint<LightState> _mqttEndpoint;
 };
 ```
 
 You can re-configure the pub/sub topics at runtime as required:
 
 ```cpp
-_mqttPubSub.configureBroker("homeassistant/light/desk_lamp/set", "homeassistant/light/desk_lamp/state");
+_mqttEndpoint.configureBroker("homeassistant/light/desk_lamp/set", "homeassistant/light/desk_lamp/state");
 ```
 
 The demo project allows the user to modify the MQTT topics via the UI so they can be changed without re-flashing the firmware.
 
+## Event Socket
+
+Beside RESTful HTTP Endpoints the Event Socket System provides a convenient communication path between the client and the ESP32. It uses a single WebSocket connection to synchronize state and to push realtime data to the client. The client needs to subscribe to the topics he is interested. Only clients who have an active subscription will receive data. Every authenticated client may make use of this system as the security settings are set to `AuthenticationPredicates::IS_AUTHENTICATED`.
+
+### Emit an Event
+
+The Event Socket provides an overloaded `emit()` function to push data to all subscribed clients. This is used by various esp32sveltekit classes to push real time data to the client. First an event must be registered with the Event Socket by calling `_socket.registerEvent("CustomEvent");`. Only then clients may subscribe to this custom event and you're entitled to emit event data:
+
+```cpp
+void emit(String event, String payload);
+void emit(const char *event, const char *payload);
+void emit(const char *event, const char *payload, const char *originId, bool onlyToSameOrigin = false);
+```
+
+The latter function allowing a selection of the recipient. If `onlyToSameOrigin = false` the payload is distributed to all subscribed clients, except the `originId`. If `onlyToSameOrigin = true` only the client with `originId` will receive the payload. This is used by the [EventEndpoint](#event-socket-endpoint) to sync the initial state when a new client subscribes.
+
+### Receive an Event
+
+A callback or lambda function can be registered to receive an ArduinoJSON object and the originId of the client sending the data:
+
+```cpp
+_socket.onEvent("CostumEvent",[&](JsonObject &root, int originId)
+{
+  bool ledState = root["led_on"];
+});
+```
+
+### Get Notified on Subscriptions
+
+Similarly a callback or lambda function may be registered to get notified when a client subscribes to an event:
+
+```cpp
+_socket.onEvent("CostumEvent",[&](const String &originId, bool sync)
+{
+  Serial.println("New Client subscribed: " + originId);
+});
+```
+
+The boolean parameter provided will always be `true`.
+
+### Push Notifications to All Clients
+
+It is possibly to send push notifications to all clients by using the Event Socket. These will be displayed as toasts an the client side. Either directly call
+
+```cpp
+esp32sveltekit.getSocket()->pushNotification("Pushed a message!", PUSHINFO);
+```
+
+or keep a local pointer to the `EventSocket` instance. It is possible to send `PUSHINFO`, `PUSHWARNING`, `PUSHERROR` and `PUSHSUCCESS` events to all clients.
+
 ## Security features
 
 The framework has security features to prevent unauthorized use of the device. This is driven by [SecurityManager.h](https://github.com/theelims/ESP32-sveltekit/blob/main/lib/framework/SecurityManager.h).
@@ -397,24 +477,6 @@ esp32sveltekit.recoveryMode();
 
 will force a start of the AP regardless of the AP settings. It will not change the the AP settings. To exit the recovery mode restart the device or change the AP settings in the UI.
 
-### Push Notifications to All Clients
-
-It is possibly to send push notifications to all clients by using Server Side Events. These will be displayed as toasts an the client side. Either directly call
-
-```cpp
-esp32sveltekit.getNotificationEvents()->pushNotification("Pushed a message!", PUSHINFO, millis());
-```
-
-or keep a local pointer to the `NotificationEvents` instance. It is possible to send `PUSHINFO`, `PUSHWARNING`, `PUSHERROR` and `PUSHSUCCESS` events to all clients. The HTTP endpoint for this service is at `/events/notifications`.
-
-In addition the raw `send()` function is mapped out as well:
-
-```cpp
-esp32sveltekit.getNotificationEvents()->send("Pushed a message!", "event", millis());
-```
-
-This allows you to send your own Server-Sent Events without opening a new HTTP connection.
-
 ### Power Down with Deep Sleep
 
 This API service can place the ESP32 in the lowest power deep sleep mode consuming only a few µA. It uses the EXT1 wakeup source, so the ESP32 can be woken up with a button or from a peripherals interrupt. Consult the [ESP-IDF Api Reference](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/sleep_modes.html#_CPPv428esp_sleep_enable_ext1_wakeup8uint64_t28esp_sleep_ext1_wakeup_mode_t) which GPIOs can be used for this. The RTC will also be powered down, so an external pull-up or pull-down resistor is required. It is not possible to persist variable state through the deep sleep. To optimize the deep sleep power consumption it is advisable to use the callback function to put pins with external pull-up's or pull-down's in a special isolated state to prevent current leakage. Please consult the [ESP-IDF Api Reference](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/sleep_modes.html#configuring-ios-deep-sleep-only) for this.
@@ -441,7 +503,7 @@ esp32sveltekit.getSleepService()->sleepNow();
 
 ### Battery State of Charge
 
-A small helper class let's you update the battery icon in the status bar. This is useful if you have a battery operated IoT device. It must be enabled in [features.ini](https://github.com/theelims/ESP32-sveltekit/blob/main/features.ini). It uses Server-sent events and exposes two functions that can be used to update the clients.
+A small helper class let's you update the battery icon in the status bar. This is useful if you have a battery operated IoT device. It must be enabled in [features.ini](https://github.com/theelims/ESP32-sveltekit/blob/main/features.ini). It uses the [Event Socket](#event-socket) and exposes two functions that can be used to update the clients.
 
 ```cpp
 esp32sveltekit.getBatteryService()->updateSOC(float stateOfCharge); // update state of charge in percent (0 - 100%)