test(ci): verify CI after merging changes into fork main #2736
+63
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…av03/fory into remove-commented-content
test(ci): merge remove-commented-content into fork main
|
I have a question: It seems that the commit logs in Fory’s history are mostly PR descriptions and not related to the local commits. In your actions, does |
|
@rajuyadav03 you should submit a pr to https://github.com/rajuyadav03/fory/tree/main to verify whether it works |
|
@rajuyadav03 you should create a PR in your forked repo, and merge the PR into your forked main branch to see whether it works instead of create a PR to fory. From the current PR code, I don't think it will work. You just run a pipeline to update local history of checked repo, it won't take effect after the PR got merged |
…ort se/de in multi-thread (apache#2737) ## What does this PR do? 1. Remove the lifetime annotations from `WriteContext` and `ReadContext` because managing their lifetimes was too troublesome. The **trade-off** is that, previously, `MetaWriterResolver` held references to the corresponding `TypeDef` obtained from `TypeResolver`, and both contexts also fetched references to `Harness` objects managed by `TypeResolver`. Now, all these references have been replaced with `clone()` to simplify lifetime management. This may introduce a non-negligible performance overhead, but it was the only practical solution I could implement.🥲 2. Remove `&fory` from the member variables of `Context`. Instead, `&fory` is now passed in as a function parameter rather than being retrieved via `context.get_fory()`. This change modified many API signatures. 3. Add two pools to `Fory`’s member variables to allow reuse of both types of contexts. Tests confirmed that the same context addresses were reused multiple times. However, since automatic recycling would require `Arc<Fory>`, only manual recycling has been implemented so far — this operation is handled internally(within `serialize()/deserialize()`), and users don’t need to recycle contexts manually. Also, for the `de/serialize_with_context(context)` functions, if users call them directly, the user's manually managed contexts will **not** be returned to the pool. 4. Add `reset()` methods to be executed on `Context objects` before recycling. 5. Modified `TypeResolver::sorted_field_names_map` from `RefCell<...>` to `RwLock<...>`, and changed `MetaReaderResolver::reading_type_defs` from `Vec<Rc<TypeMeta>>` to `Vec<Arc<TypeMeta>>`. Split `Fory::MetaStringResolver` to `ReadContext::MetaStringResolver` and `WriteContext::MetaStringResolver`. In addition, `RefReader` was unsafely marked as `Send` and `Sync`. These changes allow `Fory` to support serialization across multiple threads. A concrete example can be found in `rust/tests/tests/test_multi_thread.rs`. However, I’m not sure whether using `Lock` and `Arc` will impact single-thread performance. But it may be troublesome to write another `ThreadSafeFory`. 6. The reason why I `unsafe impl Send/Sync for RefReader {}`, is that , `RefReader` contains `Box<dyn Any>`, which by default is not `Send`/`Sync`. But In our usage, each ref_reader is only accessed within a context, by `one` thread at a time, so it is safe to implement `Send`/`Sync` manually using `unsafe`. ## Related issues - close apache#2717 ## Does this PR introduce any user-facing change? yes, like implementing in EXT, need to pass a extra parameter `fory`.
### Problem Fixes apache#2742 Python CI downloads Bazel binary on every run, which: - Takes 2-5 seconds per run - Can fail due to transient network errors - Wastes bandwidth and CI resources ### Solution Implemented GitHub Actions caching for Bazel binary to avoid repeated downloads. ### Changes - `.github/workflows/ci.yml`: Added actions/cache@v4 step to cache Bazel binary - Caches ~/bin/bazel and ~/.local/bin/bazel - Cache key includes OS, architecture, and Bazel version hash - Invalidates cache when Bazel version changes - `ci/tasks/common.py`: Updated install_bazel() function - Checks if cached Bazel binary exists before downloading - Verifies cached binary works by running bazel --version - Automatically re-downloads if cached binary is corrupted - Skips download entirely when cache is valid ### Testing Tested all scenarios: - Fresh install (no cache) - downloads successfully - Cache hit (valid binary) - skips download, saves time - Corrupted cache - detects corruption and recovers automatically - All Python syntax and YAML validation passed ### Benefits - Faster builds: Saves 2-5 seconds per CI run when cache hits - More reliable: Reduces dependency on network availability - Cost savings: Less bandwidth usage and shorter CI runtime ### Related Follow-up to apache#2733 (retry logic for Bazel downloads) Fixes apache#2742
<!-- **Thanks for contributing to Apache Fory™.** **If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fory™** community has requirements on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). - Apache Fory™ has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? <!-- Describe the details of this PR. --> ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
…d code using compile-time fields sort algorithm (apache#2749) ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? - designed a new fields sort algorithm which is friendly to compile-time languages, the compile-time languages can use this new fields sort algorithm to generate serialize code at compile time - rewrite fory rust derive macro for smaller and faster generated code Given struct: ```rust #[derive(ForyObject, Debug, PartialEq)] struct Person1 { f1: Color1, f2: Color1, // skip f3: Color2, f5: Vec<Color1>, f6: Option<Color1>, f7: Option<Color1>, f8: Color1, last: i8, } ``` For following struct, this PR generates code: ```rust fn fory_write_data( &self, fory: &fory_core::fory::Fory, context: &mut fory_core::resolver::context::WriteContext, is_field: bool, ) { fory_core::serializer::write_ref_info_data::< i8, >(&self.last, fory, context, true, true, false); fory_core::serializer::write_ref_info_data::< Vec<Color1>, >(&self.f5, fory, context, true, false, false); fory_core::serializer::write_ref_info_data::< Color1, >(&self.f1, fory, context, true, false, false); fory_core::serializer::write_ref_info_data::< Color1, >(&self.f2, fory, context, true, false, false); fory_core::serializer::write_ref_info_data::< Color2, >(&self.f3, fory, context, true, false, false); fory_core::serializer::write_ref_info_data::< Option<Color1>, >(&self.f6, fory, context, true, false, false); fory_core::serializer::write_ref_info_data::< Option<Color1>, >(&self.f7, fory, context, true, false, false); fory_core::serializer::write_ref_info_data::< Color1, >(&self.f8, fory, context, true, false, false); } fn fory_read_data( fory: &fory_core::fory::Fory, context: &mut fory_core::resolver::context::ReadContext, is_field: bool, ) -> Result<Self, fory_core::error::Error> { let _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; let _f5 = fory_core::serializer::read_ref_info_data::< Vec<Color1>, >(fory, context, true, false, false)?; let _f1 = fory_core::serializer::read_ref_info_data::< Color1, >(fory, context, true, false, false)?; let _f2 = fory_core::serializer::read_ref_info_data::< Color1, >(fory, context, true, false, false)?; let _f3 = fory_core::serializer::read_ref_info_data::< Color2, >(fory, context, true, false, false)?; let _f6 = fory_core::serializer::read_ref_info_data::< Option<Color1>, >(fory, context, true, false, false)?; let _f7 = fory_core::serializer::read_ref_info_data::< Option<Color1>, >(fory, context, true, false, false)?; let _f8 = fory_core::serializer::read_ref_info_data::< Color1, >(fory, context, true, false, false)?; Ok(Self { last: _last, f5: _f5, f1: _f1, f2: _f2, f3: _f3, f6: _f6, f7: _f7, f8: _f8, }) } ``` This PR also reverts apache#2724 since it generats lots of inefficient code and bloat code size ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
<!-- **Thanks for contributing to Apache Fory™.** **If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fory™** community has requirements on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). - Apache Fory™ has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? <!-- Describe the details of this PR. --> ## Related issues apache#2750 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? add maven cache to ci for faster build ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
<!-- **Thanks for contributing to Apache Fory™.** **If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fory™** community has requirements on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). - Apache Fory™ has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? <!-- Describe the details of this PR. --> ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? <!-- Describe the details of this PR. --> ## Related issues apache#2752 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
<!-- **Thanks for contributing to Apache Fory™.** **If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fory™** community has requirements on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). - Apache Fory™ has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? <!-- Describe the details of this PR. --> ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
<!-- **Thanks for contributing to Apache Fory™.** **If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fory™** community has requirements on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). - Apache Fory™ has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? <!-- Describe the details of this PR. --> ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
<!-- **Thanks for contributing to Apache Fory™.** **If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fory™** community has requirements on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). - Apache Fory™ has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? <!-- Describe the details of this PR. --> ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
…field id (apache#2758) ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? This pr implemented a new fory_read_compatible macro function to use match by assigned field id for better performance. - When read type meta, assign field id based on local field type info. If type info are consistent(excluding nullability), then assign remote field to local field. Otherwise, skip this remote field value when deserializing, use default value instead. - When generating fory_read_compatible code, use match by field id arm, which will compiled to switch. This is much more efficient compared to previous compare field by string - Skip using Option is local field is primitive and not Option. Now the macro generate much more efficient and compact code: ```rust fn fory_read_compatible( fory: &fory_core::fory::Fory, context: &mut fory_core::resolver::context::ReadContext, ) -> Result<Self, fory_core::error::Error> { let remote_type_id = context.reader.read_varuint32(); let meta_index = context.reader.read_varuint32(); let meta = context.get_meta(meta_index as usize); let fields = { let meta = context.get_meta(meta_index as usize); meta.get_field_infos().clone() }; let mut _f7: i16 = 0 as i16; let mut _f5: i8 = 0 as i8; let mut _last: i8 = 0 as i8; let mut _f4: Option<String> = None; let mut _f3: Option<Vec<i8>> = None; let mut _f6: Option<Vec<i16>> = None; let mut _f1: Option<HashMap<i8, Vec<i8>>> = None; let local_type_def = fory .get_type_resolver() .get_type_info(std::any::TypeId::of::<Self>()) .get_type_def(); let high_bytes = &local_type_def[..8]; let local_type_hash = i64::from_le_bytes(high_bytes.try_into().unwrap()); if meta.get_hash() == local_type_hash { <Self as fory_core::serializer::Serializer>::fory_read_data( fory, context, false, ) } else { for _field in fields.iter() { match _field.field_id { 0i16 => { if !&_field.field_type.nullable { _f7 = fory_core::serializer::read_ref_info_data::< i16, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _f7 = <i16 as fory_core::serializer::ForyDefault>::fory_default(); } else { _f7 = fory_core::serializer::read_ref_info_data::< i16, >(fory, context, true, true, false)?; } } } 1i16 => { if !&_field.field_type.nullable { _f5 = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _f5 = <i8 as fory_core::serializer::ForyDefault>::fory_default(); } else { _f5 = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } } } 2i16 => { if !&_field.field_type.nullable { _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _last = <i8 as fory_core::serializer::ForyDefault>::fory_default(); } else { _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } } } 3i16 => { if !&_field.field_type.nullable { _f4 = Some( fory_core::serializer::read_ref_info_data::< String, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f4 = Some( <String as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f4 = Some( fory_core::serializer::read_ref_info_data::< String, >(fory, context, true, false, false)?, ); } } } 4i16 => { if !&_field.field_type.nullable { _f3 = Some( fory_core::serializer::read_ref_info_data::< Vec<i8>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f3 = Some( <Vec< i8, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f3 = Some( fory_core::serializer::read_ref_info_data::< Vec<i8>, >(fory, context, true, false, false)?, ); } } } 5i16 => { if !&_field.field_type.nullable { _f6 = Some( fory_core::serializer::read_ref_info_data::< Vec<i16>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f6 = Some( <Vec< i16, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f6 = Some( fory_core::serializer::read_ref_info_data::< Vec<i16>, >(fory, context, true, false, false)?, ); } } } 6i16 => { if !&_field.field_type.nullable { _f1 = Some( fory_core::serializer::read_ref_info_data::< HashMap<i8, Vec<i8>>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f1 = Some( <HashMap< i8, Vec<i8>, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f1 = Some( fory_core::serializer::read_ref_info_data::< HashMap<i8, Vec<i8>>, >(fory, context, true, false, false)?, ); } } } _ => { let field_type = &_field.field_type; let read_ref_flag = fory_core::serializer::skip::get_read_ref_flag( &field_type, ); fory_core::serializer::skip::skip_field_value( fory, context, &field_type, read_ref_flag, ) .unwrap(); } } } Ok(Self { f7: _f7, f5: _f5, last: _last, f4: _f4.unwrap_or_default(), f3: _f3.unwrap_or_default(), f6: _f6.unwrap_or_default(), f1: _f1.unwrap_or_default(), }) } ``` ## Related issues apache#2492 apache#2545 Closes apache#2761 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
…pache#2763) ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? use compatible bool isntead enum to simplify API: ```rust let fory = Fory::default().compatible(true); ``` ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? - [x] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
…tion (apache#2764) ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? <!-- Describe the details of this PR. --> ## Related issues Closes apache#2759 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
## What does this PR do? Introduce alternate "compact" row encoding that better uses knowledge of fixed-size types and sacrifices alignment to save space. Introduce new Builder pattern to avoid explosion of `Encoders` static methods as more features are added to row format. Optimizations include: * struct stores fixed-size fields (e.g. Int128. FixedSizeBinary) inline in fixed-data area without offset + size * struct of all fixed-sized fields is itself considered fixed-size to store in other struct or array * struct skips null bitmap if all fields are non-nullable * struct sorts fields by fixed-size for best-effort (but not guaranteed) alignment * struct can use less than 8 bytes for small data (int, short, etc) * struct null bitmap stored at end of struct to borrow alignment padding if possible * array stores fixed-size fields inline in fixed-data area without offset+size * array header uses 4 bytes for size (since Collection and array are only int-sized) and leaves remaining 4 bytes for start of null bitmap Fixups include: * toString better handles varbinary / fixed-binary (hex dump of first 256 bytes) * start making Javadoc for row format Compromises: * less alignment could increase access time, but this is opt-in. and I think on modern processors it is not such a big deal. * increased complexity of offset lookup, try to pre-compute in an array when possible and use `StableValue` when it is GA Not compatible with existing row format. ## Related issues Fixes apache#2337 ## Does this PR introduce any user-facing change? New API for new Compact codec. Existing codec unchanged.
<!-- **Thanks for contributing to Apache Fory™.** **If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fory™** community has requirements on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). - Apache Fory™ has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## Why? <!-- Describe the purpose of this PR. --> There is no test of pointer field for meta share mode before, so I created some and fixed bugs regard that ## What does this PR do? <!-- Describe the details of this PR. --> Add more test to refine meta share mode. - PointerFields: test serialization and deserialization of pointer fields - PointerFieldsInconsistent: test serialization and deserialization of Inconsistent pointer fields. - ComplexRoundTrip: more complex object to test metashare mode, contains interface type field. There are some bug fix: - field order problem introduce by apache#2749, re-enable the go ci - add a runtime value param to readTypeInfo function, so we can choose pointer or value to read with based on the runtime-value. ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> apache#2192 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [x] Does this PR introduce any public API change? no - [x] Does this PR introduce any binary protocol compatibility change? no ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
…ect/assert/unwrap with Result (apache#2765) ## What does this PR do? - simply refactor `error.rs` - replace `panic/expect/unwrap` with `Result` - replace `assert` with `ensure` - change Serialize api to return `Result` ## Related issues close apache#2706 ## Does this PR introduce any user-facing change? 1. user should use `unwrap/?` when se/de. 2. when they customize serializer, will return `Result`. - [x] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? --------- Co-authored-by: chaokunyang <[email protected]>
## Why? Simplify the serialization interface for better ergonomics API ## What does this PR do? - Remove `Fory` from `Serializer` trait - Make `WriteContext` as only enviroment for serialization - Make `ReadContext` as only enviroment for deserialization - Use `OnceLock` for thread-safe, lazily initialization for `Pool` ## Related issues apache#2737 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
…2766) ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? - support optional typehint for dataclass fields - fastpath for numeric and string fields serialization ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
<!-- **Thanks for contributing to Apache Fory™.** **If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fory™** community has requirements on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). - Apache Fory™ has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? <!-- Describe the details of this PR. --> ## Related issues apache#2192 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [x] Does this PR introduce any public API change? no - [x] Does this PR introduce any binary protocol compatibility change? no ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
## What does this PR do? This PR rewrite whole rust serializer to provide a much more clean, dynamic and flexible serializer system. Main changes inlucde: - New Serializer trait inferface: privide new `fory_write`/`fory_write_generic`/`fory_read`/`fory_read_with_type_info` API,, to allow control nested serialization behaviour - Provide `fory_static_type_id` to allow fast compile-time serializer dispatch - Implement fory collection/map dynamic xlang serialization protocol, make it work with trait object and shared/circular reference - Removed `is_field` parameter, it inotroduce lots of confusion to users - Rewrite whole error processing system, use static method instead, and support dump backtrace in apache#2780 - Add struct field read/write hook, support dump field write/read info and support register customized debug hook - Refactored the derive macro with better generated code - Move `fory_read_compatible` to `StrucSerializer` ## Related issues - Closes apache#2777 - Closes apache#2775 - Closes apache#2780 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? add rust thread safety and troubleshooting doc ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
apache#2782) ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? use rc instead of arc for type meta for faster performance ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
## Why? tests is more idiomatic to AI coding ## What does this PR do? rename fory-tests to tests ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
apache#2784) ## Why? <!-- Describe the purpose of this PR. --> support dataclass compatible mode for python native mode <!-- Describe the details of this PR. --> ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
…ache#2787) ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? <!-- Describe the details of this PR. --> ## Related issues Closes apache#2720 apache#2719 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? - fix rust map read - fix java map read - enable rust xlang tests ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
… code (apache#2779) <!-- **Thanks for contributing to Apache Fory™.** **If this is your first time opening a PR on fory, you can refer to [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).** Contribution Checklist - The **Apache Fory™** community has requirements on the naming of pr titles. You can also find instructions in [CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). - Apache Fory™ has a strong focus on performance. If the PR you submit will have an impact on performance, please benchmark it first and provide the benchmark result here. --> ## Why? The previous field ordering algorithm mixed different field types (strings, lists, maps, etc.) into broad groups that were difficult to maintain and reason about at compile time. The new specification provides a clearer categorization that: 1. Improves compile-time predictability: Separates fields into distinct groups (primitives, other internal types, lists, sets, maps, others) with clear sorting rules for each group 2. Enhances cross-language compatibility: Aligns with the updated xlang serialization specification for consistent behavior across all Fory language implementations 3. Simplifies code generation: Each field group has its own simple sorting rule, making the generated code more maintainable and easier to verify ## What does this PR do? Modified `fory/codegen/utils.go`: * Rewrote sortFields() function to implement the new 6-group categorization * Added getFieldGroup() function to categorize fields into their respective sorting groups * Added field group constants (groupPrimitive, groupOtherInternalType, groupList, groupSet, groupMap, groupOther) * Each group now has its own clear sorting logic as specified in the xlang serialization spec ## Related issues More details see in pr apache#2749 <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. --> Co-authored-by: Shawn Yang <[email protected]> Co-authored-by: Pan Li <[email protected]>
…2789) ## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? make type meta resolve return type info directly to avoid uncesssary copy ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
rajuyadav03
requested review from
PragmaTwice,
chaokunyang and
theweipeng
as code owners
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Testing CI; this is a verification PR for #2715