Skip to content

Conversation

EvandroLG
Copy link
Contributor

@EvandroLG EvandroLG commented Oct 2, 2025

This PR adds support for the queue_url parameter in S3 ClickPipes. It enables event-based continuous ingestion using SQS queue notifications. Files are ingested immediately based on S3 events rather than through lexicographical polling.

Changes

  • Added queue_url field to API models, resource models, and schema with SQS URL validation
  • Added validation: requires is_continuous = true, S3 storage type, and IAM authentication
  • Added examples

@EvandroLG EvandroLG self-assigned this Oct 2, 2025
@EvandroLG EvandroLG requested a review from Copilot October 2, 2025 09:55
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for SQS Queue URL configuration in the S3 object storage source for ClickHouse continuous ingestion. This enables event-based file ingestion using S3 notifications instead of lexicographical scanning.

  • Adds QueueURL field to the object storage source model with validation
  • Implements configuration validation to ensure queue_url is only used with S3, continuous mode, and authentication
  • Updates the schema with proper regex validation for SQS URL format

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
pkg/resource/models/clickpipe_resource.go Adds QueueURL field to ClickPipeObjectStorageSourceModel struct and corresponding object type/value methods
pkg/resource/clickpipe.go Implements schema definition with validation, plan modification logic, and state synchronization for queue_url
pkg/internal/api/clickpipe_models.go Adds QueueURL field to the API model for object storage source

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@EvandroLG EvandroLG requested a review from Copilot October 2, 2025 12:14
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@EvandroLG EvandroLG requested a review from Copilot October 2, 2025 14:57
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@EvandroLG EvandroLG changed the title WIP: support SQS Queue URL for continuous ingestion in S3 Support SQS Queue URL for continuous ingestion in S3 Oct 2, 2025
@EvandroLG EvandroLG requested a review from a team October 2, 2025 15:21
@EvandroLG EvandroLG marked this pull request as ready for review October 2, 2025 15:22
@EvandroLG EvandroLG requested a review from a team as a code owner October 2, 2025 15:22
@EvandroLG EvandroLG force-pushed the feat/unordered-mode branch from 6f30b55 to d6737e2 Compare October 3, 2025 10:07
@EvandroLG EvandroLG requested a review from whites11 October 3, 2025 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants