Implement `Allocative` for the protocol views. #4687

MathieuDutSik · 2025-09-29T13:51:30Z

Motivation

Tracking the size of the objects being used is important for considering the memory usage and resolving the OOM.

Proposal

The Allocative trait is implemented for the views, and the code is followed from there.

Several design decisions were made:

For the Weighted distribution, the entries are dropped.
In the views, the context is not taken into account of memory usage.
For the keys of Ed25519, Secp256k1, EvmSecp256k1, the sizes are written explicitly.

Test Plan

The CI.
More tests need to be added for tracking the memory usage in the operations.

Release Plan

The goal is to add it to the TestNet Conway.

Links

None.

deuszx · 2025-09-29T16:45:54Z

linera-chain/src/certificate/generic.rs

+        visitor.visit_field(Key::new("GenericCertificate_value"), &self.value);
+        visitor.visit_field(Key::new("GenericCertificate_round"), &self.round);
+        for (public_key, _) in &self.signatures {
+            visitor.visit_field(Key::new("ValidatorPublicKey"), public_key);


public keys also have a fixed size, don't they? Why assume the size for signatures but not public keys?

You are right. Corrected.

afck

I think we should be able to use derive a bit more.

I'm not sure an Allocative implementation should acquire locks. It's probably meant to be very fast. Anyway, if we do try_read, we should never panic if we don't get the lock.

afck · 2025-10-21T11:59:51Z

linera-base/src/crypto/mod.rs

+            }
+        }
+    }
+}


Would something like this work?

#[derive(Allocative)] pub enum AccountPublicKey { Ed25519(#[allocative(visit = visit_allocative_simple)] ed25519::Ed25519PublicKey), } fn visit_allocative_simple<'a, 'b, T>(_: &T, visitor: &'a mut Visitor<'b>) { visitor.visit_simple_sized::<T>(); }

Thanks, that works!
How do you find this?

linera-base/src/crypto/mod.rs

linera-views/src/views/collection_view.rs

linera-views/src/views/hashable_wrapper.rs

linera-views/src/views/key_value_store_view.rs

linera-views/src/views/queue_view.rs

linera-views/src/views/reentrant_collection_view.rs

afck · 2025-10-23T08:09:37Z

linera-base/src/crypto/mod.rs

+/// The SECP256k1 public key sizes also used for validators.
+pub const SECP256K1_PUBLIC_KEY_SIZE: usize = std::mem::size_of::<secp256k1::Secp256k1PublicKey>();
+/// The SECP256k1 signature sizes also used for validators.


Suggested change

/// The SECP256k1 public key sizes also used for validators.

pub const SECP256K1_PUBLIC_KEY_SIZE: usize = std::mem::size_of::<secp256k1::Secp256k1PublicKey>();

/// The SECP256k1 signature sizes also used for validators.

/// The secp256k1 public key sizes also used for validators.

pub const SECP256K1_PUBLIC_KEY_SIZE: usize = std::mem::size_of::<secp256k1::Secp256k1PublicKey>();

/// The secp256k1 signature sizes also used for validators.

(I think that's how we spell it elsewhere.)

afck · 2025-10-23T08:13:09Z

linera-base/src/crypto/mod.rs

+/// The SECP256k1 public key sizes also used for validators.
+pub const SECP256K1_PUBLIC_KEY_SIZE: usize = std::mem::size_of::<secp256k1::Secp256k1PublicKey>();
+/// The SECP256k1 signature sizes also used for validators.
+pub const SECP256K1_SIGNATURE_SIZE: usize = std::mem::size_of::<secp256k1::Secp256k1Signature>();
+


Actually, do we still need those at all? Aren't those "simple", too?

(And even if: If it's only used in one place I'd rather use size_of there directly?)

afck · 2025-10-23T08:15:23Z

linera-chain/src/certificate/lite.rs

+        let signatures = self.signatures.deref();
+        for (public_key, _) in signatures {
+            visitor.visit_field(Key::new("ValidatorPublicKey"), public_key);
+            visitor.visit_simple(Key::new("ValidatorSignature"), 64);


Can we use size_of here instead of a number?

afck · 2025-10-23T08:16:43Z

linera-chain/src/certificate/lite.rs

    pub signatures: Cow<'a, [(ValidatorPublicKey, ValidatorSignature)]>,
 }

+impl Allocative for LiteCertificate<'_> {


Is this necessary because Allocative doesn't support Cow?

afck · 2025-10-23T08:17:02Z

linera-chain/src/certificate/lite.rs

+    fn visit<'a, 'b: 'a>(&self, visitor: &'a mut Visitor<'b>) {
+        visitor.visit_field(Key::new("LiteCertificate_value"), &self.value);
+        visitor.visit_field(Key::new("LiteCertificate_round"), &self.round);
+        let signatures = self.signatures.deref();


I wonder if we should only count it if its owned?

afck · 2025-10-23T08:17:59Z

linera-chain/src/data_types/mod.rs

    pub signature: ValidatorSignature,
 }

+impl<T: Allocative> Allocative for Vote<T> {


Looks like we should be able to derive this, and either use visit_allocative_simple or derive Allocative for ValidatorSignature, too?

afck · 2025-10-23T08:20:35Z

linera-views/src/views/hashable_wrapper.rs

    _phantom: PhantomData<C>,
+    #[allocative(visit = visit_allocative_simple)]
    stored_hash: Option<O>,
+    #[allocative(visit = visit_allocative_simple)]


(That will only count the stack size of the mutex, rather than its contents, will it? It would be more correct to add size_of::<Option<O>>() to that. Since it's just a hash, maybe it doesn't matter much, but I'd at least add a short comment.

To be clear: In either case I agree that we do not need to acquire the mutex lock here!)

afck · 2025-10-23T08:20:48Z

linera-views/src/views/key_value_store_view.rs

    sizes: ByteMapView<C, u32>,
+    #[allocative(visit = visit_allocative_simple)]
    stored_hash: Option<HasherOutput>,
+    #[allocative(visit = visit_allocative_simple)]


(Same here.)

deuszx reviewed Sep 29, 2025

View reviewed changes

MathieuDutSik force-pushed the allocative_view branch 2 times, most recently from af9011c to 4772d6c Compare September 30, 2025 13:06

MathieuDutSik marked this pull request as ready for review September 30, 2025 14:35

MathieuDutSik force-pushed the allocative_view branch from 3155f45 to c6e8523 Compare October 7, 2025 13:24

afck reviewed Oct 21, 2025

View reviewed changes

MathieuDutSik force-pushed the allocative_view branch from 3eeb6a9 to f81a32f Compare October 22, 2025 18:48

Introduce the Allocative to the views.

2341ef4

MathieuDutSik force-pushed the allocative_view branch from f81a32f to 2341ef4 Compare October 22, 2025 18:50

Somw corrections after rebase.

e08c132

afck reviewed Oct 23, 2025

View reviewed changes

Implement Allocative for the protocol views. #4687

Are you sure you want to change the base?

Implement Allocative for the protocol views. #4687

Conversation

MathieuDutSik commented Sep 29, 2025

Motivation

Proposal

Test Plan

Release Plan

Links

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afck left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Implement `Allocative` for the protocol views. #4687

Implement `Allocative` for the protocol views. #4687