Skip to content

feat: implement fake DMQ node #2635

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 38 commits into
base: main
Choose a base branch
from
Draft

feat: implement fake DMQ node #2635

wants to merge 38 commits into from
+2,071 −231

Conversation

jpraynaud
Copy link
Member

@jpraynaud jpraynaud commented Jul 8, 2025 •
edited
Loading

Content

Note

This is a work in progress

This PR includes the implementation of a fake DMQ node in the relay:

  • N2C local message submission mini-protocol: implementation of the server side of the protocol (and refactoring of the client side).
  • N2C local message notification mini-protocol: implementation of the server side of the protocol (and refactoring of the client side).
  • N2N message submission mini-protocol: fake implementation with libp2p in the relay.
  • Integration tests for the N2C mini-protocols.
  • E2E test modification to support diffusion via the fake DMQ node.
  • Infrastructure modification to support the fake DQ node

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • CHANGELOG file is updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • All check jobs of the CI have succeeded
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested
  • Documentation
    • Update documentation website (if relevant)
    • No new TODOs introduced

Issue(s)

Closes #2627

@jpraynaud jpraynaud self-assigned this Jul 8, 2025
github-advanced-security[bot]
github-advanced-security bot found potential problems Jul 8, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 8, 2025 •
edited
Loading

Test Results

0 files   -     4  0 suites   - 158   0s ⏱️ - 22m 49s
0 tests  - 2 152  0 ✅  - 2 152  0 💤 ±0  0 ❌ ±0 
0 runs   - 6 602  0 ✅  - 6 602  0 💤 ±0  0 ❌ ±0 

Results for commit 657a484. ± Comparison against base commit 7ec438a.

♻️ This comment has been updated with latest results.

@jpraynaud jpraynaud requested a review from Copilot July 8, 2025 16:03
Copilot
Copilot AI reviewed Jul 8, 2025
View reviewed changes
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds support for a Decentralized Message Queue (DMQ) node across end-to-end tests, relay components, signers, aggregators, and the mithril-dmq crate.

  • Introduce DMQ_NODE_SOCKET_PATH env var and socket setup in end-to-end signer and aggregator.
  • Extend relay components and CLI commands to accept DMQ socket path and Cardano network settings.
  • Refactor mithril-dmq crate to split client/server interfaces and enable future_dmq by default.

Reviewed Changes

Copilot reviewed 42 out of 43 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
mithril-test-lab/.../src/mithril/signer.rs Insert DMQ_NODE_SOCKET_PATH setup in Signer
mithril-test-lab/.../src/mithril/relay_signer.rs Add DMQ socket and network env in RelaySigner
mithril-test-lab/.../src/mithril/relay_aggregator.rs Add DMQ socket and network env in RelayAggregator
mithril-test-lab/.../src/mithril/aggregator.rs Insert DMQ socket env in end-to-end Aggregator
mithril-relay/src/relay/signer.rs Wire up DMQ publisher server in SignerRelay
mithril-relay/src/relay/aggregator.rs Wire up DMQ consumer server in AggregatorRelay
internal/mithril-dmq/src/publisher/client/pallas.rs Rename to client interface and update implementations
Comments suppressed due to low confidence (1)

mithril-test-lab/mithril-end-to-end/src/mithril/signer.rs:99

  • [nitpick] End-to-end tests don’t currently cover the DMQ_NODE_SOCKET_PATH setup. Add tests that exercise DMQ-based workflows to verify this new configuration is applied correctly.
        // TODO: make this configurable

@jpraynaud jpraynaud force-pushed the jpraynaud/2627-fake-dmq-node-in-relay branch from 730ee75 to 3761142 Compare July 15, 2025 16:46
github-advanced-security[bot]
github-advanced-security bot found potential problems Jul 15, 2025
View reviewed changes
@jpraynaud jpraynaud force-pushed the jpraynaud/2627-fake-dmq-node-in-relay branch 2 times, most recently from 14815a4 to ac101c4 Compare July 16, 2025 10:52
github-advanced-security[bot]
github-advanced-security bot found potential problems Jul 16, 2025
View reviewed changes
@jpraynaud jpraynaud force-pushed the jpraynaud/2627-fake-dmq-node-in-relay branch from ac101c4 to 054bcb5 Compare July 16, 2025 12:43
@jpraynaud jpraynaud force-pushed the jpraynaud/2627-fake-dmq-node-in-relay branch from b603cb0 to 59f3dc6 Compare July 16, 2025 13:44
@jpraynaud jpraynaud temporarily deployed to testing-preview July 16, 2025 13:57 — with GitHub Actions Inactive
@jpraynaud jpraynaud temporarily deployed to dev-follower-preview July 16, 2025 14:41 — with GitHub Actions Inactive
@jpraynaud jpraynaud force-pushed the jpraynaud/2627-fake-dmq-node-in-relay branch from 59f3dc6 to 57452a5 Compare July 22, 2025 13:48
github-advanced-security[bot]
github-advanced-security bot found potential problems Jul 22, 2025
View reviewed changes
mithril-relay/src/p2p/peer.rs
Comment on lines +82 to +83
let bytes =
bincode::serde::encode_to_vec(self, bincode::config::standard()).map_err(|e| e.into());

Check warning

Code scanning / clippy

returning the result of a let binding from a block Warning

returning the result of a let binding from a block
mithril-relay/src/p2p/peer.rs
Comment on lines +82 to +83
let bytes =
bincode::serde::encode_to_vec(self, bincode::config::standard()).map_err(|e| e.into());

Check warning

Code scanning / clippy

returning the result of a let binding from a block Warning

returning the result of a let binding from a block
mithril-relay/src/p2p/peer.rs
let bytes =
bincode::serde::encode_to_vec(self, bincode::config::standard()).map_err(|e| e.into());

bytes

Check warning

Code scanning / clippy

returning the result of a let binding from a block Warning

returning the result of a let binding from a block
mithril-relay/src/p2p/peer.rs
let bytes =
bincode::serde::encode_to_vec(self, bincode::config::standard()).map_err(|e| e.into());

bytes

Check warning

Code scanning / clippy

returning the result of a let binding from a block Warning

returning the result of a let binding from a block
@jpraynaud jpraynaud force-pushed the jpraynaud/2627-fake-dmq-node-in-relay branch from 57452a5 to 900fb4e Compare August 18, 2025 16:43
github-advanced-security[bot]
github-advanced-security bot found potential problems Aug 18, 2025
View reviewed changes
internal/mithril-dmq/src/consumer/server/pallas.rs
logger: Logger,
}

impl DmqConsumerServerPallas {

Check warning

Code scanning / clippy

method drop_server is never used Warning

method drop\_server is never used
internal/mithril-dmq/src/consumer/server/pallas.rs
}

/// Drops the current `DmqServer`, if it exists.
async fn drop_server(&self) -> StdResult<()> {

Check warning

Code scanning / clippy

method drop_server is never used Warning

method drop\_server is never used
github-advanced-security[bot]
github-advanced-security bot found potential problems Aug 18, 2025
View reviewed changes
internal/mithril-dmq/src/consumer/server/pallas.rs
logger: Logger,
}

impl DmqConsumerServerPallas {

Check warning

Code scanning / clippy

method drop_server is never used Warning

method drop\_server is never used
internal/mithril-dmq/src/consumer/server/pallas.rs
}

/// Drops the current `DmqServer`, if it exists.
async fn drop_server(&self) -> StdResult<()> {

Check warning

Code scanning / clippy

method drop_server is never used Warning

method drop\_server is never used
internal/mithril-dmq/src/consumer/server/pallas.rs
}
},
None => {
return Err(anyhow!("DMQ message receiver is not registered"));

Check warning

Code scanning / clippy

unneeded return statement Warning

unneeded return statement
internal/mithril-dmq/src/consumer/server/pallas.rs
}
},
None => {
return Err(anyhow!("DMQ message receiver is not registered"));

Check warning

Code scanning / clippy

unneeded return statement Warning

unneeded return statement
jpraynaud added 28 commits August 19, 2025 11:32
@jpraynaud
feat(dmq): add 'DmqConsumerServerPallas' implementation of 'DmqConsum…
f9c05df 
…erServer'
@jpraynaud
feat(relay): add support for DMQ messages
f510273
@jpraynaud
feat(relay): update passive relay for DMQ messages
b7e49ce
@jpraynaud
feat(relay): update signer relay for DMQ messages
cf081a3
@jpraynaud
feat(relay): update signer command for DMQ messages
507f8cd
@jpraynaud
feat(relay): update aggregator relay for DMQ messages
c7ce6e1
@jpraynaud
feat(relay): update aggregator command for DMQ messages
6e73fef
@jpraynaud
feat(relay): update integration test for DMQ messages
afcf385
@jpraynaud
feat(dmq): export DMQ servers for publisher and consumer
f1b7074
@jpraynaud
feat(relay): use binary encoding for exchanging messages in P2P pubsu…
1d19768 
…b topics
@jpraynaud
fix(dmq): activate Pallas server sides for Unix only
8b7bffc
@jpraynaud
fix(dmq): add missing 'kes_period' in 'DmqMsg'
b28bc67
@jpraynaud
fix(dmq): gate code behind 'future_dmq' feature
b5a17dc
@jpraynaud
fix(dmq): missing wait for Done message in publisher server state mac…
89684f8 
…hine
@jpraynaud
fix(relay): fix clippy warning
371ce7f
@jpraynaud
feat(infra): support for DMQ protocol in aggregator infrastructure
a308087
@jpraynaud
feat(infra): support for DMQ protocol in signer infrastructure
a87a8bd
@jpraynaud
feat(ci): support for using DMQ in infrastucture deployment
2f85b68
@jpraynaud
fix: rebase from main branch
126d6cf 
'mithril_common::test_utils' has been renamed to 'mithril_common::test'.
@jpraynaud
wip(dmq): temporary fix KES signature
950fa54
@jpraynaud
fix(dmq): run receive/serve messages on separate threads
9774ded
@jpraynaud
test(dmq): add integration test for publisher client/server
5b4a6f1
@jpraynaud
test(dmq): add integration test for consumer client/server
bb2dff2
@jpraynaud
fix(dmq): consumer can not serve messages multiple times
0c3e23f
@jpraynaud
wip(e2e): support for fake DMQ node
bac64bb
@jpraynaud
wip: DO NOT MERGE - activate 'future_dmq'
81b82a8
@jpraynaud
wip: DO NOT MERGE - activate 'allow_skip_signer_certification'
ae39f4c
@jpraynaud
wip: DO NOT MERGE - deactivate signer HTTP signature broadcast
657a484
@jpraynaud jpraynaud force-pushed the jpraynaud/2627-fake-dmq-node-in-relay branch from 900fb4e to 657a484 Compare August 20, 2025 15:50
github-advanced-security[bot]
github-advanced-security bot found potential problems Aug 20, 2025
View reviewed changes
internal/mithril-dmq/tests/consumer_client_server.rs
async move {
let consumer_client = DmqConsumerClientPallas::<DmqMessageTestPayload>::new(
socket_path,
cardano_network.clone(),

Check warning

Code scanning / clippy

using clone on type CardanoNetwork which implements the Copy trait Warning test

using clone on type CardanoNetwork which implements the Copy trait
internal/mithril-dmq/tests/consumer_client_server.rs
slog_scope::logger(),
);
let mut messages = vec![];
signature_dmq_tx.send(create_fake_msg(b"msg_1").await.into()).unwrap();

Check warning

Code scanning / clippy

useless conversion to the same type: mithril_dmq::DmqMessage Warning test

useless conversion to the same type: mithril\_dmq::DmqMessage
internal/mithril-dmq/tests/consumer_client_server.rs
);
let mut messages = vec![];
signature_dmq_tx.send(create_fake_msg(b"msg_1").await.into()).unwrap();
signature_dmq_tx.send(create_fake_msg(b"msg_2").await.into()).unwrap();

Check warning

Code scanning / clippy

useless conversion to the same type: mithril_dmq::DmqMessage Warning test

useless conversion to the same type: mithril\_dmq::DmqMessage
internal/mithril-dmq/tests/consumer_client_server.rs
signature_dmq_tx.send(create_fake_msg(b"msg_1").await.into()).unwrap();
signature_dmq_tx.send(create_fake_msg(b"msg_2").await.into()).unwrap();
messages.extend_from_slice(&consumer_client.consume_messages().await.unwrap());
signature_dmq_tx.send(create_fake_msg(b"msg_3").await.into()).unwrap();

Check warning

Code scanning / clippy

useless conversion to the same type: mithril_dmq::DmqMessage Warning test

useless conversion to the same type: mithril\_dmq::DmqMessage
internal/mithril-dmq/tests/publisher_client_server.rs
.set_ttl(100);
let publisher_client = DmqPublisherClientPallas::<DmqMessageTestPayload>::new(
socket_path,
cardano_network.clone(),

Check warning

Code scanning / clippy

using clone on type CardanoNetwork which implements the Copy trait Warning test

using clone on type CardanoNetwork which implements the Copy trait
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@jpraynaud jpraynaud

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Implement a fake DMQ node with Mithril relay
1 participant
@jpraynaud