-
Notifications
You must be signed in to change notification settings - Fork 14
Support SQS Queue URL for continuous ingestion in S3 #388
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds support for SQS Queue URL configuration in the S3 object storage source for ClickHouse continuous ingestion. This enables event-based file ingestion using S3 notifications instead of lexicographical scanning.
- Adds
QueueURL
field to the object storage source model with validation - Implements configuration validation to ensure queue_url is only used with S3, continuous mode, and authentication
- Updates the schema with proper regex validation for SQS URL format
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
File | Description |
---|---|
pkg/resource/models/clickpipe_resource.go | Adds QueueURL field to ClickPipeObjectStorageSourceModel struct and corresponding object type/value methods |
pkg/resource/clickpipe.go | Implements schema definition with validation, plan modification logic, and state synchronization for queue_url |
pkg/internal/api/clickpipe_models.go | Adds QueueURL field to the API model for object storage source |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
6f30b55
to
d6737e2
Compare
This PR adds support for the
queue_url
parameter in S3 ClickPipes. It enables event-based continuous ingestion using SQS queue notifications. Files are ingested immediately based on S3 events rather than through lexicographical polling.Changes
queue_url
field to API models, resource models, and schema with SQS URL validationis_continuous = true
, S3 storage type, and IAM authentication