feat: use `clarity-serialization` in `clarity` #6310

Jiloc · 2025-07-21T14:50:10Z

Description

Points for Discussion from #6297:

Error handling: In clarity-serialization, I consolidated all error types into a single CodecError enum with a 1:1 match with the original errors in clarity. However, when converting back to the older error types using From<CodecError>, some variants don’t have a clear or meaningful equivalent. I’ve routed them through generic catch-all cases. Even if the current code doesn't reach those conversions, I am not happy with this mapping strategy and am open to suggestions!
Retro-compatibility of ClarityName and ContractName: The guarded_string! macro for these types now returns the new CodecError instead of a RuntimeErrorType.
Feel free to throw any other topic you think would be worth discussing!

Applicable issues

fixes #

Additional info (benefits, drawbacks, caveats)

Checklist

Test coverage for new or modified code paths
Changelog is updated
Required documentation changes (e.g., docs/rpc/openapi.yaml and rpc-endpoints.md for v2 endpoints, event-dispatcher.md for new events)
New clarity functions have corresponding PR in clarity-benchmarking repo
New integration test(s) added to bitcoin-tests.yml

…end-on-clarity-serialization

…ity-depend-on-clarity-serialization

jcnelson · 2025-08-04T19:47:05Z

clarity/src/vm/types/signatures.rs

+    ) -> Result<TupleTypeSignature> {
+        if let SymbolicExpressionType::List(ref name_type_pairs) = type_def.expr {
+            let mapped_key_types = parse_name_type_pairs(epoch, name_type_pairs, accounting)?;
+            TupleTypeSignature::try_from(mapped_key_types).map_err(CheckErrors::from)


Please revert this error conversion

done in b0957e3

jcnelson · 2025-08-04T19:47:30Z

clarity/src/vm/types/signatures.rs

        }
    }
 }

-/// Parsing functions.
-impl TypeSignature {
+pub trait TypeSignatureExt {


Please add a docstring explaining this trait.

Actually, seeing this massive trait and the traits above it, all of which only seem to exist because SymbolicExpression isn't in clarity-serialization, I'm inclined to take back what I said earlier about keeping SymbolicExpression in clarity. Please instead move SymbolicExpression and its ilk to clarity-serialization so these traits can be removed. We'll just port the feature flags from clarity to clarity-serialization.

I added a comment for now, but I am open to revisit this if you think otherwise! The main reasons I initially avoided moving this in clarity-serialization was because most of these functions require both for SymbolicExpression and CostTracker which I saw more as a runtime construct and more appropriate for clarity. CostTracker also depends on ClarityCostFunction, ExecutionCost, and CostErrors.

But with the latest changes I already needed to move ExecutionCost and CostErrors into clarity-serialization, so I’m open to also moving ClarityCostFunction and CostTracker if you think that would be cleaner. Let me know what you prefer!

jcnelson · 2025-08-04T19:48:53Z

stackslib/src/blockstack_cli.rs

@@ -236,6 +238,22 @@ impl From<ClarityError> for CliError {
    }
 }

+impl From<ClarityCodecError> for CliError {


Please drop

done in 40a95ee

jcnelson · 2025-08-04T19:49:07Z

stackslib/src/net/mod.rs

@@ -537,6 +537,12 @@ impl From<InterpreterError> for Error {
    }
 }

+impl From<CodecError> for Error {


Please drop

done in 40a95ee

jcnelson

Thank you for your continued work on separating out Clarity serialization code from clarity. There are a few show-stopping issues that must be addressed, which are detailed in my comments, but I'll summarize below.

Error types in Clarity are generally consensus-critical. In particular, InterpreterError and CheckErrors are handled differently in transaction-processing -- the former marks a block as invalid, while the latter can be mined. The replacement of both of these variants and others with CodecErrors is a consensus-critical bug that must be reverted throughout.
There is a lot of code churn here that has arisen from the fact that SymbolicExpression is in clarity, not clarity-serialization. This has lead to the need for several traits whose only purpose is to allow types defined in clarity-serialization to continue to have their existing methods which either took SymbolicExpression-based types as arguments or returned a SymbolicExpression-based type. This code churn can and should be avoided, since all of this is consensus-critical. I think it would be better to move SymbolicExpression to clarity-serialization, even though it would mean that feature flags in clarity would need to be duplicated in clarity-serialization.

Jiloc · 2025-08-06T16:30:32Z

clarity-serialization/src/types/mod.rs

+#[derive(Clone, PartialEq, Eq, Hash, Debug, Serialize, Deserialize)]
+pub struct FunctionIdentifier {
+    identifier: String,
+}
+


This is needed for the StackTrace type used in Error

Jiloc · 2025-08-06T16:31:26Z

clarity-serialization/src/types/serialization.rs

+    IOError(IncomparableError<std::io::Error>),
+    BadTypeError(CheckErrors),
+    DeserializationError(String),
+    DeserializeExpected(Box<TypeSignature>),


originally, this was DeserializeExpected(TypeSignature) does this change cause any problems?

I think this is fine, because AFAIK we never match on whether or not a substructure is a Box<T> anywhere in the codebase (that would be very weird).

Can you explain the reason for the change?

Same reason as for #6310 (comment). But if you prefer I could move even this change to the separate PR

Jiloc · 2025-08-06T16:38:36Z

clarity-serialization/Cargo.toml

@@ -12,11 +12,12 @@ readme = "README.md"
 [dependencies]
 lazy_static = { workspace = true }
 regex = { version = "1", default-features = false }
+rusqlite = { workspace = true, optional = true }


needed for implementing ToSql and FromSql for ExecutionCost, and defining InterpreterError::SqliteError(IncomparableError<SqliteError>)

Hmm, this might need some thought. I thought clarity-serialization was supposed to be able to run in WASM? If so, then rusqlite can't be a required dependency IIRC.

It shouldn't be a problem in this case, rustqlite will only be included if built with the rusqlite feature. If building this with

cargo build -p clarity-serialization --features "wasm-web" (for js support)
cargo build -p clarity-serialization --features "wasm-deterministic" (for deterministic environment like CosmWasm)

it will still work!

clarity-serialization/Cargo.toml

clarity/src/vm/analysis/errors.rs

Jiloc · 2025-08-06T16:53:40Z

clarity/src/vm/types/signatures.rs

All the structs remaining in this file (e.g. FunctionSignature, FixedFunction, etc.) and their implementations, were not originally needed in clarity-serialization, but since we are going to rename this into clarity-types I was thinking of moving them all. Any opinions?

Jiloc · 2025-08-06T17:12:00Z

clarity/src/vm/types/signatures.rs

        }
    }
 }

-/// Parsing functions.
-impl TypeSignature {
+pub trait TypeSignatureExt {


I added a comment for now, but I am open to revisit this if you think otherwise! The main reasons I initially avoided moving this in clarity-serialization was because most of these functions require both for SymbolicExpression and CostTracker which I saw more as a runtime construct and more appropriate for clarity. CostTracker also depends on ClarityCostFunction, ExecutionCost, and CostErrors.

But with the latest changes I already needed to move ExecutionCost and CostErrors into clarity-serialization, so I’m open to also moving ClarityCostFunction and CostTracker if you think that would be cleaner. Let me know what you prefer!

Jiloc · 2025-08-06T19:12:02Z

clarity-serialization/src/errors/analysis.rs

+pub type CheckResult<T> = Result<T, CheckError>;
+
+#[derive(Debug, PartialEq)]
+pub enum CheckErrors {


would it be consensus breaking if we modify CheckErrors to Box all TypeSignature and Value in order to remove #![allow(clippy::result_large_err)]?

I think that should be fine, because we don't ever check a type's substructure to see if it contains a Box<T>.

Given it will be require a big amount of line changes, I'll just do it in a separate PR

Just for completeness, here is the PR that boxes the data in some errors and fixes all the clippy::result_large_err warnings. No need to review it now of course :)

moodmosaic

We've opened a PR as a follow-up to expand input coverage further, in line with #6355.

Jiloc added 4 commits July 21, 2025 15:45

use clarity-serializaton in clarity

83ae404

use CodecError instead of SerializationError

96dddb8

add From<ClarityCodecError> for ClientError

59d9bf2

add CodecError conversions for stackslib

cfa3cf6

Jiloc added this to the 3.2.0.0.0 milestone Jul 21, 2025

Jiloc self-assigned this Jul 21, 2025

Jiloc requested review from a team as code owners July 21, 2025 14:50

Jiloc added this to Stacks Core Eng Jul 21, 2025

Jiloc marked this pull request as draft July 21, 2025 14:51

cargo fmt

04a9323

Jiloc moved this to Status: 💻 In Progress in Stacks Core Eng Jul 21, 2025

Jiloc modified the milestones: 3.2.0.0.0, 3.2.0.0.1 Jul 21, 2025

Jiloc added 10 commits July 21, 2025 16:33

Merge branch 'feat/clarity-serialization-crate' into feat/clarity-dep…

3eabeb4

…end-on-clarity-serialization

update Cargo.lock

1f1c190

fix cli tests

7848406

Merge branch 'feat/clarity-serialization-crate' into feat/clarity-dep…

738bea8

…end-on-clarity-serialization

remove hashbrown

526cec1

Merge branch 'feat/clarity-serialization-crate' into feat/clarity-dep…

1944b20

…end-on-clarity-serialization

remove hashbrown from stackslib

47cfbfe

cargo format

9475370

Merge branch 'feat/clarity-serialization-crate' into feat/clarity-dep…

dd8ad84

…end-on-clarity-serialization

Merge branch 'feat/clarity-serialization-crate' into feat/clarity-dep…

228bdee

…end-on-clarity-serialization

Jiloc mentioned this pull request Jul 23, 2025

feat: enable clarity-serialization in cargo workspace #6326

Merged

5 tasks

Jiloc added 5 commits July 23, 2025 17:09

Merge branch 'feat/move-clarity-serialization-to-root' into feat/clar…

05931c8

…ity-depend-on-clarity-serialization

add testing utils for conversion

5462de1

cargo fmt

28036bc

Merge branch 'feat/move-clarity-serialization-to-root' into feat/clar…

9660ae2

…ity-depend-on-clarity-serialization

Merge branch 'develop' into feat/clarity-depend-on-clarity-serialization

1c05004

jcnelson reviewed Aug 4, 2025

View reviewed changes

jcnelson requested changes Aug 4, 2025

View reviewed changes

github-project-automation bot moved this from Status: In Review to Status: 💻 In Progress in Stacks Core Eng Aug 4, 2025

Jiloc added 3 commits August 6, 2025 16:13

drop CodecError in favor of original clarity errors

40a95ee

clippy

b0957e3

remove old file

fa6d7d7

Jiloc commented Aug 6, 2025

View reviewed changes

Jiloc added 5 commits August 6, 2025 17:45

remove unnecessary map_err

aef2fe9

add comment for parsing traits

a1090c4

revert to original implementation of parse_*_type_repr

454d7e0

remove redundant clarity-serialization dependency with testing flag

15ae223

fix blockstack_cli tests

7b42ca5

Jiloc commented Aug 6, 2025

View reviewed changes

Jiloc moved this from Status: 💻 In Progress to Status: In Review in Stacks Core Eng Aug 6, 2025

Jiloc requested review from jcnelson and fdefelici August 6, 2025 19:03

Jiloc commented Aug 6, 2025

View reviewed changes

Jiloc added 3 commits August 6, 2025 20:14

add missing copyrights

9eabf67

Merge branch 'develop' into feat/clarity-depend-on-clarity-serialization

69a1941

add clarity to cargo check for wasm32

abccbe9

Jiloc requested a review from a team as a code owner August 7, 2025 21:20

Jiloc mentioned this pull request Aug 7, 2025

fix: result_large_err clippy warning #6358

Draft

5 tasks

re-add rustqlite in testing

4a35628

moodmosaic reviewed Aug 15, 2025

View reviewed changes

moodmosaic mentioned this pull request Aug 15, 2025

Expand input coverage with fuzzing #6355

Open

feat: use clarity-serialization in clarity #6310

Are you sure you want to change the base?

feat: use clarity-serialization in clarity #6310

Conversation

Jiloc commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Applicable issues

Additional info (benefits, drawbacks, caveats)

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jiloc Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jiloc Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcnelson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jiloc Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jiloc Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jiloc Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

moodmosaic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

feat: use `clarity-serialization` in `clarity` #6310

feat: use `clarity-serialization` in `clarity` #6310

Jiloc commented Jul 21, 2025 •

edited

Loading

Jiloc Aug 6, 2025 •

edited

Loading

Jiloc Aug 6, 2025 •

edited

Loading

Jiloc Aug 6, 2025 •

edited

Loading

Jiloc Aug 6, 2025 •

edited

Loading

Jiloc Aug 6, 2025 •

edited

Loading