diff --git a/CIPs/cip-123.md b/CIPs/cip-123.md new file mode 100644 index 0000000..db9a7b8 --- /dev/null +++ b/CIPs/cip-123.md @@ -0,0 +1,96 @@ +--- +cip: 123 +title: Batch pubsub updates +author: Joel Thorstensson +discussions-to: https://forum.ceramic.network/t/batch-pubsub-updates +status: Draft +category: Standards +type: Core +created: 2023-02-01 +--- + +## Simple Summary + +Introduce a new pubsub message type to batch multiple stream updates into a single pubsub message. + + +## Abstract + +This spec introduces a batch update pubsub message that creates update batches based on the `sep` value in the header of init events in Ceramic streams. Nodes that are interested in a particular `sep` value can listen for these messages and apply all tips from the batch at once. + + +## Motivation + +Ceramic under heavy write load will currently create a lot of traffic in the pubsub topic that is pure overhead for other nodes that don't care about any of the updated streams. Batching updates can be used to minimize this load. + + +## Specification + + +### Sending a batch update + +For all streams where the *Init event* has a `sep` field in the header batch requests into a single pubsub message according to the following algorithm: + +1. When a new event is added to a stream, get the `separator-value` by looking up the header field defined by the `separator-key` found in the value of the `sep` field + 1. If no other update for this `separator-value`, store the streamid and tip along with a timestamp of the current time + 2. If other updates exist already for this `separator-value` store the new streamid, tip touple along with the previous updates +2. When `T` seconds have passed since the first was created add all (streamid, tip) values to IPFS and send a `BATCH_UPDATE` message containing the CID of this data to pubsub +3. Remove the temporary values + +### Pubsub message type + +Introduce a new pubsub message type called `BATCH_UPDATE`, which has the following format: + +```js +{ + typ: 3, + sep: "/", + batch: +} +``` + +#### Batch format + +```ipldsch +type streamid Bytes +type tip Link + +type Batch struct { + tips: { streamid : [tip] } +} +``` + +### Receiving batch update + +If a node recieves a `BATCH_UPDATE` pubsub message, it follows the following algorithm: + +1. If the node is not interested in the `separator-key` / `separator-value` pair, do nothing +2. If the node is interested in the `separator-key` / `separator-value` pair, get the `batch-CID` from the message +3. Load the content of the `batch-CID` over bitswap +4. Get all tips from the batch data and apply them to the corresponding streams + + +## Rationale + +The `sep` field is currently being used in ComposeDB for Model and Model Instance Document stream types. This would enable ComposeDB to batch stream updates by Model. However, by using `sep` this solution is not tied specifically to how ComposeDB uses the Ceramic protocol. Any application could define their own `sep` and thus `separator-key` and `separator-value`. + +The `T` constant determines how quickly nodes will publish and recieve updates from other nodes. For reasonable update velocity this should be set to `1`. + + +## Backwards Compatibility + +Nodes that adopt this new batch format would need to stop sending `UPDATE` type messages. This means that any node that relies on the old update message format would be unable to hear about updates. This change is being considered in the context of ComposeDB where the (`separator-key`, `separator-value`) pair correspnds to (`"model"`, ``). ComposeDB will index data by model, so nodes would at least receive updates at every new on-chain anchor. Therefore, nodes that don't update to support the new message type can still keep in sync, only much slower. + + +## Implementation + +No implementation yet. + + +## Security Considerations + +This CIP doesn't introduce any new security considerations beyond the current pubsub system. The batch will need to be loaded from IPFS bitswap, and from a security perspective this is similar to loading events given tips of an `UPDATE` message. + + +## Copyright +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).