Skip to content

out_s3: Add parquet compression type with pure C #10691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

out_s3: Add parquet compression type with pure C #10691

wants to merge 8 commits into from
+196 −15

Conversation

cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented Aug 5, 2025 •
edited by coderabbitai bot
Loading

With apache arrow glib parquet library, we're able to support parquet format on out_s3.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    trace
    HTTP_Server  Off
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name dummy
    Tag  dummy.local
    dummy {"boolean": false, "int": 1, "long": 1, "float": 1.1, "double": 1.1, "bytes": "foo", "string": "foo"}

[OUTPUT]
    Name  s3
    Match dummy*
    Region us-east-2
    bucket fbit-parquet-s3
    Use_Put_object true
    compression parquet
    # No need to specify schema
  • Debug log output from testing the change
Fluent Bit v4.1.0
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____ 
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/ 


[2025/08/05 17:27:09] [ info] Configuration:
[2025/08/05 17:27:09] [ info]  flush time     | 5.000000 seconds
[2025/08/05 17:27:09] [ info]  grace          | 5 seconds
[2025/08/05 17:27:09] [ info]  daemon         | 0
[2025/08/05 17:27:09] [ info] ___________
[2025/08/05 17:27:09] [ info]  inputs:
[2025/08/05 17:27:09] [ info]      dummy
[2025/08/05 17:27:09] [ info] ___________
[2025/08/05 17:27:09] [ info]  filters:
[2025/08/05 17:27:09] [ info] ___________
[2025/08/05 17:27:09] [ info]  outputs:
[2025/08/05 17:27:09] [ info]      s3.0
[2025/08/05 17:27:09] [ info] ___________
[2025/08/05 17:27:09] [ info]  collectors:
[2025/08/05 17:27:09] [ info] [fluent bit] version=4.1.0, commit=0afb495f86, pid=81424
[2025/08/05 17:27:09] [debug] [engine] coroutine stack size: 36864 bytes (36.0K)
[2025/08/05 17:27:09] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/08/05 17:27:09] [ info] [simd    ] NEON
[2025/08/05 17:27:09] [ info] [cmetrics] version=1.0.5
[2025/08/05 17:27:09] [ info] [ctraces ] version=0.6.6
[2025/08/05 17:27:09] [ info] [input:dummy:dummy.0] initializing
[2025/08/05 17:27:09] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2025/08/05 17:27:09] [debug] [dummy:dummy.0] created event channels: read=26 write=27
[2025/08/05 17:27:09] [debug] [s3:s3.0] created event channels: read=28 write=29
[2025/08/05 17:27:09] [debug] [tls] attempting to load certificates from system keychain of macOS
# <snip of loading certificates logs>
[2025/08/05 17:27:09] [debug] [tls] finished loading keychain certificates, total loaded: 153
[2025/08/05 17:27:09] [debug] [aws_credentials] Initialized Env Provider in standard chain
[2025/08/05 17:27:09] [debug] [aws_credentials] creating profile (null) provider
[2025/08/05 17:27:09] [debug] [aws_credentials] Initialized AWS Profile Provider in standard chain
[2025/08/05 17:27:09] [debug] [aws_credentials] Not initializing EKS provider because AWS_ROLE_ARN was not set
[2025/08/05 17:27:09] [debug] [aws_credentials] Not initializing ECS/EKS HTTP Provider because AWS_CONTAINER_CREDENTIALS_RELATIVE_URI and AWS_CONTAINER_CREDENTIALS_FULL_URI is not set
[2025/08/05 17:27:09] [debug] [aws_credentials] Initialized EC2 Provider in standard chain
[2025/08/05 17:27:09] [debug] [aws_credentials] Sync called on the EC2 provider
[2025/08/05 17:27:09] [debug] [aws_credentials] Init called on the env provider
[2025/08/05 17:27:09] [ info] [output:s3:s3.0] Sending locally buffered data from previous executions to S3; buffer=/tmp/fluent-bit/s3/fbit-parquet-s3
[2025/08/05 17:27:09] [ info] [output:s3:s3.0] Pre-compression chunk size is 2394, After compression, chunk is 2384 bytes
[2025/08/05 17:27:10] [debug] [upstream] KA connection #34 to s3.us-east-2.amazonaws.com:443 is connected
[2025/08/05 17:27:10] [debug] [http_client] not using http_proxy for header
[2025/08/05 17:27:10] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/08/05 17:27:10] [debug] [upstream] KA connection #34 to s3.us-east-2.amazonaws.com:443 is now available
[2025/08/05 17:27:10] [debug] [output:s3:s3.0] PutObject http status=200
[2025/08/05 17:27:10] [ info] [output:s3:s3.0] Successfully uploaded object /fluent-bit-logs/dummy.local/2025/08/05/08/27/09-objectt33OX2sM
[2025/08/05 17:27:10] [debug] [aws_credentials] upstream_set called on the EC2 provider
[2025/08/05 17:27:10] [ info] [output:s3:s3.0] worker #0 started
[2025/08/05 17:27:10] [ info] [sp] stream processor started
[2025/08/05 17:27:10] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/08/05 17:27:15] [debug] [task] created task=0x1137040a0 id=0 OK
[2025/08/05 17:27:15] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/08/05 17:27:15] [debug] [output:s3:s3.0] Creating upload timer with frequency 60s
[2025/08/05 17:27:15] [debug] [out flush] cb_destroy coro_id=0
[2025/08/05 17:27:15] [debug] [task] destroy task=0x1137040a0 (task_id=0)
[2025/08/05 17:27:18] [engine] caught signal (SIGTERM)
[2025/08/05 17:27:18] [ info] [input] pausing dummy.0
[2025/08/05 17:27:18] [ info] [output:s3:s3.0] thread worker #0 stopping...
[2025/08/05 17:27:18] [ info] [output:s3:s3.0] thread worker #0 stopped
[2025/08/05 17:27:18] [ info] [output:s3:s3.0] Sending all locally buffered data to S3
[2025/08/05 17:27:18] [ info] [output:s3:s3.0] Pre-compression chunk size is 504, After compression, chunk is 1904 bytes
[2025/08/05 17:27:18] [debug] [output:s3:s3.0] PutObject http status=200
[2025/08/05 17:27:18] [ info] [output:s3:s3.0] Successfully uploaded object /fluent-bit-logs/dummy.local/2025/08/05/08/27/15-objectkoBRRSAl
  • Attached Valgrind output that shows no leaks or memory corruption was found

With leaks on macOS, there's no leaks:

Process:         fluent-bit [81424]
Path:            /Users/USER/*/fluent-bit
Load Address:    0x100f58000
Identifier:      fluent-bit
Version:         0
Code Type:       ARM64
Platform:        macOS
Parent Process:  leaks [81423]
Target Type:     live task

Date/Time:       2025-08-05 17:27:19.085 +0900
Launch Time:     2025-08-05 17:27:09.515 +0900
OS Version:      macOS 15.5 (24F74)
Report Version:  7
Analysis Tool:   /usr/bin/leaks

Physical footprint:         15.6M
Physical footprint (peak):  18.8M
Idle exit:                  untracked
----

leaks Report Version: 4.0, multi-line stacks
Process 81424: 3008 nodes malloced for 428 KB
Process 81424: 0 leaks for 0 total leaked bytes.

[2025/08/05 17:27:19] [engine] caught signal (SIGCONT)
[2025/08/05 17:27:19] Fluent Bit Dump

With valgrind:

==361658== LEAK SUMMARY:
==361658==    definitely lost: 0 bytes in 0 blocks
==361658==    indirectly lost: 0 bytes in 0 blocks
==361658==      possibly lost: 0 bytes in 0 blocks
==361658==    still reachable: 68,984 bytes in 677 blocks
==361658==         suppressed: 0 bytes in 0 blocks
==361658== Reachable blocks (those to which a pointer was found) are not shown.

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

  • New Features

    • Added support for Parquet compression in the S3 output plugin, allowing users to choose "parquet" compression when Apache Arrow and Parquet libraries are enabled.
    • Updated configuration options and documentation to include "parquet" alongside "gzip" and "arrow" as supported compression types.
    • Enhanced build and CI workflows to detect and enable Apache Parquet support automatically when dependencies are available.

  • Bug Fixes

    • Improved validation and error messages to ensure proper configuration when using "arrow" and "parquet" compression types.

@coderabbitai coderabbitai
Copy link

coderabbitai bot commented Aug 5, 2025 •
edited
Loading

Walkthrough

This change introduces conditional support for Apache Parquet compression in the AWS S3 output plugin, leveraging Arrow GLib and Parquet GLib if available. It adds CMake detection for Parquet, new constants and functions for Parquet compression, and updates the S3 plugin to recognize and handle the "parquet" compression type alongside existing options.

Changes

Cohort / File(s) Change Summary
CMake Parquet Detection
CMakeLists.txt, src/aws/compression/arrow/CMakeLists.txt		 Adds CMake logic to detect Parquet GLib support and conditionally enable Parquet-related includes, linking, and preprocessor macros.
Compression Type Constants
include/fluent-bit/aws/flb_aws_compress.h		 Adds new constant FLB_AWS_COMPRESS_PARQUET for Parquet compression.
S3 Output Plugin Compression Handling
plugins/out_s3/s3.c		 Updates compression logic to support "parquet" as a valid type, adjusts validation, error messages, and documentation accordingly.
Arrow/Parquet Compression Implementation
src/aws/compression/arrow/compress.c, src/aws/compression/arrow/compress.h		 Adds Parquet compression implementation and declaration, including a new function to convert JSON to Parquet format, with robust error handling.
Compression Options Table
src/aws/flb_aws_compress.c		 Registers the new "parquet" compression option in the compression options array, conditionally compiled if Parquet support is detected.
CI Workflow Update
.github/workflows/unit-tests.yaml		 Adds a build matrix entry enabling Arrow support and installs Apache Arrow and Parquet GLib development packages on Ubuntu runners.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant S3Plugin
    participant AWSCompress
    participant ArrowCompress
    participant ParquetGLib

    User->>S3Plugin: Configure compression = "parquet"
    S3Plugin->>AWSCompress: Request compression function for "parquet"
    AWSCompress->>ArrowCompress: Call out_s3_compress_parquet(json, size)
    ArrowCompress->>ParquetGLib: Parse JSON to Arrow Table
    ArrowCompress->>ParquetGLib: Convert Arrow Table to Parquet buffer
    ParquetGLib-->>ArrowCompress: Return Parquet buffer
    ArrowCompress-->>AWSCompress: Return compressed buffer and size
    AWSCompress-->>S3Plugin: Return compressed buffer
    S3Plugin-->>User: Uploads Parquet-compressed data to S3
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~18 minutes

Poem

In the world of clouds, a bunny hops,
Now Parquet joins the S3 drops!
With Arrow feathers and Parquet might,
Compression options take new flight.
CMake sniffs, the build grows wise,
And fluffy code now multiplies.
🐇✨

Note

⚡️ Unit Test Generation is now available in beta!

Learn more here, or try it out under "Finishing Touches" below.

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 67295b6 and f5b2741.

📒 Files selected for processing (8)
  • .github/workflows/unit-tests.yaml (3 hunks)
  • CMakeLists.txt (1 hunks)
  • include/fluent-bit/aws/flb_aws_compress.h (1 hunks)
  • plugins/out_s3/s3.c (10 hunks)
  • src/aws/compression/arrow/CMakeLists.txt (1 hunks)
  • src/aws/compression/arrow/compress.c (2 hunks)
  • src/aws/compression/arrow/compress.h (1 hunks)
  • src/aws/flb_aws_compress.c (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • src/aws/flb_aws_compress.c
🚧 Files skipped from review as they are similar to previous changes (7)
  • .github/workflows/unit-tests.yaml
  • src/aws/compression/arrow/compress.c
  • src/aws/compression/arrow/CMakeLists.txt
  • include/fluent-bit/aws/flb_aws_compress.h
  • src/aws/compression/arrow/compress.h
  • CMakeLists.txt
  • plugins/out_s3/s3.c
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (29)
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
  • GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
  • GitHub Check: pr-compile-centos-7
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
  • GitHub Check: PR - fuzzing test
✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch cosmo0920-implement-parquet-comression-with-apache-arrow-glib-parquet
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@cosmo0920 cosmo0920 force-pushed the cosmo0920-implement-parquet-comression-with-apache-arrow-glib-parquet branch from 85316ab to 0afb495 Compare August 5, 2025 07:55
@cosmo0920 cosmo0920 force-pushed the cosmo0920-implement-parquet-comression-with-apache-arrow-glib-parquet branch from 243f704 to 4eaed0d Compare August 5, 2025 08:16
@cosmo0920 cosmo0920 force-pushed the cosmo0920-implement-parquet-comression-with-apache-arrow-glib-parquet branch from 4eaed0d to e69bbd2 Compare August 5, 2025 08:21
@cosmo0920 cosmo0920 marked this pull request as ready for review August 5, 2025 09:28
@cosmo0920 cosmo0920 force-pushed the cosmo0920-implement-parquet-comression-with-apache-arrow-glib-parquet branch from e7c14b8 to 8eb1493 Compare August 7, 2025 09:40
coderabbitai[bot]
coderabbitai bot reviewed Aug 7, 2025
View reviewed changes
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
.github/workflows/unit-tests.yaml (2)

55-55: -DFLB_ARROW=On is always built, but the Arrow/Parquet libs are not conditionally installed
The matrix now always attempts an Arrow build with both compilers, yet the required libraries are installed unconditionally later (Lines 94-101). This means every job – even those that do not pass -DFLB_ARROW=On – now spends extra time adding the Arrow APT source and downloading large packages.

Consider restricting the new flag to a dedicated matrix entry or gate the install step with

if: matrix.flb_option == '-DFLB_ARROW=On'

to save several minutes per CI run.

70-73: exclude block duplicates compiler filtering already implied by build flag
-DFLB_ARROW=On is excluded for clang here, but the flag is offered in the matrix only once (Line 55). If future contributors move the flag around, this silent coupling between two lists becomes error-prone.

A clearer pattern is to define dedicated strategy axes, e.g.

include:
  - flb_option: "-DFLB_ARROW=On"
    compiler:
      cc: gcc
      cxx: g++

This removes the need for paired exclude rows and keeps intent explicit.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 230815d and 8eb1493.

📒 Files selected for processing (1)
  • .github/workflows/unit-tests.yaml (3 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (25)
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
  • GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
  • GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
  • GitHub Check: pr-compile-centos-7
  • GitHub Check: PR - fuzzing test

@cosmo0920
build: Check existence of arrow-glib-parquet library
ef6da71 
Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920
aws: compression: Implement compression of Apache Arrow Parquet
665f73d 
Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920
aws: Implement a handler for parquet compression
2e37681 
Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920
out_s3: Use arrow or parquet correctly as compression methods
67dd71c 
Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920
aws: compress: Add error logs
a575eff 
Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920
build: aws: compress: Use LIBRARIES instead of LDFLAGS for safety
120a28d 
Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920
workflows: Add FLB_ARROW task for testing
258f694 
Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920
build: aws: compression: Use LDFLAGS instead of LIBRARIES
f5b2741 
Using LIBRARIES does not point into the right place of the apache
arrow-glib and parquet-glib libraries.

Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920 cosmo0920 force-pushed the cosmo0920-implement-parquet-comression-with-apache-arrow-glib-parquet branch from 67295b6 to f5b2741 Compare August 7, 2025 15:40
@sagar-18
Copy link

@cosmo0920 any update here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

@PettitWesley PettitWesley Awaiting requested review from PettitWesley PettitWesley is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@niedbalski niedbalski Awaiting requested review from niedbalski niedbalski is a code owner

@patrick-stephens patrick-stephens Awaiting requested review from patrick-stephens patrick-stephens is a code owner

@celalettin1286 celalettin1286 Awaiting requested review from celalettin1286 celalettin1286 is a code owner

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich

@koleini koleini Awaiting requested review from koleini koleini is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
Fluent Bit v4.1
Development

Successfully merging this pull request may close these issues.
3 participants
@cosmo0920 @bilalmughal @sagar-18