CIP: Batch pubsub updates

CIPs/cip-123.md

@@ -0,0 +1,96 @@
+---
+cip: 123
+title: Batch pubsub updates
+author: Joel Thorstensson <[email protected]>
+discussions-to: https://forum.ceramic.network/t/batch-pubsub-updates
+status: Draft
+category: Standards
+type: Core
+created: 2023-02-01
+---
+
+## Simple Summary
+<!--Provide a simplified and layman-accessible explanation of the CIP.-->
+Introduce a new pubsub message type to batch multiple stream updates into a single pubsub message.
+
+
+## Abstract
+<!--A short (~200 word) description of the technical issue being addressed.-->
+This spec introduces a batch update pubsub message that creates update batches based on the `sep` value in the header of init events in Ceramic streams. Nodes that are interested in a particular `sep` value can listen for these messages and apply all tips from the batch at once.
+
+
+## Motivation
+<!--Motivation is critical for CIPs that want to change the Ceramic protocol. It should clearly explain why the existing protocol specification is inadequate to address the problem that the CIP solves. CIP submissions without sufficient motivation may be rejected outright.-->
+Ceramic under heavy write load will currently create a lot of traffic in the pubsub topic that is pure overhead for other nodes that don't care about any of the updated streams. Batching updates can be used to minimize this load. 
+
+
+## Specification
+<!--The technical specification should describe the syntax and semantics of any new feature.-->
+
+### Sending a batch update
+
+For all streams where the *Init event* has a `sep` field in the header batch requests into a single pubsub message according to the following algorithm:
+
+1. When a new event is added to a stream, get the `separator-value` by looking up the header field defined by the `separator-key` found in the value of the `sep` field
+   1. If no other update for this `separator-value`, store the streamid and tip along with a timestamp of the current time
+   2. If other updates exist already for this `separator-value` store the new streamid, tip touple along with the previous updates
+2. When `T` seconds have passed since the first was created add all (streamid, tip) values to IPFS and send a `BATCH_UPDATE` message containing the CID of this data to pubsub 
+3. Remove the temporary values
+
+### Pubsub message type
+
+Introduce a new pubsub message type called `BATCH_UPDATE`, which has the following format:
+
+```js
+{
+  typ: 3,
+  sep: "<separator-key>/<separator-value>",
+  batch: <batch-CID>
+}
+```
+
+#### Batch format
+
+```ipldsch
+type streamid Bytes
+type tip Link
+
+type Batch struct {
+  tips: { streamid : [tip] }
+}
+```
+
+### Receiving batch update
+
+If a node recieves a `BATCH_UPDATE` pubsub message, it follows the following algorithm:
+
+1. If the node is not interested in the `separator-key` / `separator-value` pair, do nothing
+2. If the node is interested in the `separator-key` / `separator-value` pair, get the `batch-CID` from the message
+3. Load the content of the `batch-CID` over bitswap
+4. Get all tips from the batch data and apply them to the corresponding streams
+
+
+## Rationale
+<!--The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale may also provide evidence of consensus within the community, and should discuss important objections or concerns raised during discussion.-->
+The `sep` field is currently being used in ComposeDB for Model and Model Instance Document stream types. This would enable ComposeDB to batch stream updates by Model. However, by using `sep` this solution is not tied specifically to how ComposeDB uses the Ceramic protocol. Any application could define their own `sep` and thus `separator-key` and `separator-value`.
+
+The `T` constant determines how quickly nodes will publish and recieve updates from other nodes. For reasonable update velocity this should be set to `1`.
+
+
+## Backwards Compatibility
+<!--All CIPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The CIP must explain how the author proposes to deal with these incompatibilities. CIP submissions without a sufficient backwards compatibility section may be rejected outright.-->
+Nodes that adopt this new batch format would need to stop sending `UPDATE` type messages. This means that any node that relies on the old update message format would be unable to hear about updates. This change is being considered in the context of ComposeDB where the (`separator-key`, `separator-value`) pair correspnds to (`"model"`, `<model-streamid>`). ComposeDB will index data by model, so nodes would at least receive updates at every new on-chain anchor. Therefore, nodes that don't update to support the new message type can still keep in sync, only much slower.
+
+
+## Implementation
+<!--The implementations must be completed before any CIP is given status "Final", but it need not be completed before the CIP is accepted.-->
+No implementation yet.
+
+
+## Security Considerations
+<!--All CIPs must contain a section that discusses the security implications/considerations relevant to the proposed change. Include information that might be important for security discussions, surfaces risks and can be used throughout the life cycle of the proposal. E.g. include security-relevant design decisions, concerns, important discussions, implementation-specific guidance and pitfalls, an outline of threats and risks and how they are being addressed. CIP submissions missing the "Security Considerations" section will be rejected. An CIP cannot proceed to status "Final" without a Security Considerations discussion deemed sufficient by the reviewers.-->
+This CIP doesn't introduce any new security considerations beyond the current pubsub system. The batch will need to be loaded from IPFS bitswap, and from a security perspective this is similar to loading events given tips of an `UPDATE` message.
+
+
+## Copyright
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).