feat: add Consume task

[mady20]

Proposed task: Consume

Proposed syntax:

id: kinesis_data_streams_consumer
namespace: dev
tasks:
  - id: consume
    type: io.kestra.plugin.aws.kinesis.Consume
    streamName: required string
    startingPosition: TRIM_HORIZON | LATEST | AT_TIMESTAMP
    startingTimestamp: optional string, required if startingPosition is AT_TIMESTAMP
    maxRecords: optional integer, max records per poll
    maxRecordsTotal: optional integer, stop consuming after total records
    pollDuration: optional duration, wait between polls
    maxDuration: optional duration, stop consuming after total time
    streamArn: optional string, required if streamName not provided

Example usage:

id: kinesis_consume_example
namespace: company.team
tasks:
  - id: consume
    type: io.kestra.plugin.aws.kinesis.Consume
    accessKeyId: "AWS_ACCESS_KEY_ID"
    secretKeyId: "AWS_SECRET_ACCESS_KEY"
    sessionToken: "AWS_SESSION_TOKEN"
    region: "eu-central-1"
    streamName: "mystream"
    startingPosition: "TRIM_HORIZON"
    pollDuration: PT5S
    maxRecords: 1000

Notes:

• Requires either streamName or streamArn.
• Tracks record count and execution duration metrics.
• Includes unit and integration tests for various scenarios.

Links:

• API Refernce
• KinesisClient Java SDK
• Developing Kinesis consumers with SDK

closes #644

---

Contributor Checklist ✅

• PR Title and commits follows conventional commits
• Add a closes #ISSUE_ID or fixes #ISSUE_ID in the description if the PR relates to an opened issue.
• Documentation updated (plugin docs from @Schema for properties and outputs, @Plugin with examples, README.md file with basic knowledge and specifics).
• Setup instructions included if needed (API keys, accounts, etc.).
• Prefix all rendered properties by r not rendered (eg: rHost).
• Use runContext.logger() to log enough important infos where it's needed and with the best level (DEBUG, INFO, WARN or ERROR).

⚙️ Properties

• Properties are declared with Property<T> carrier type, do not use @PluginProperty.
• Mandatory properties must be annotated with @NotNull and checked during the rendering.
• You can model a JSON thanks to a simple Property<Map<String, Object>>.

🌐 HTTP

• Must use Kestra’s internal HTTP client from io.kestra.core.http.client

📦 JSON

• If you are serializing response from an external API, you may have to add a @JsonIgnoreProperties(ignoreUnknown = true) at the mapped class level. So that we will avoid to crash the plugin if the provider add a new field suddenly.
• Must use Jackson mappers provided by core (io.kestra.core.serializers)

✨ New plugins / subplugins

• Make sure your new plugin is configured like mentioned here.
• Add a package-info.java under each sub package respecting this format and choosing the right category.
• Icons added in src/main/resources/icons in SVG format and not in thumbnail (keep it big):
  • plugin-icon.svg
  • One icon per package, e.g. io.kestra.plugin.aws.svg
  • For subpackages, e.g. io.kestra.plugin.aws.s3, add io.kestra.plugin.aws.s3.svg
    See example here.
• Use "{{ secret('YOUR_SECRET') }}" in the examples for sensible infos such as an API KEY.
• If you are fetching data (one, many or too many), you must add a Property<FetchType> fetchType to be able to use FETCH_ONE, FETCH and even STORE to store big amount of data in the internal storage.
• Align the """ to close examples blocks with the flow id.

🧪 Tests

• Unit Tests added or updated to cover the change (using the RunContext to actually run tasks).
• Add sanity checks if possible with a YAML flow inside src/test/resources/flows.
• Avoid disabling tests for CI. Instead, configure a local environment whenever it's possible with .github/setup-unit.sh (which can be executed locally and in the CI) all along with a new docker-compose-ci.yml file (do not edit the existing docker-compose.yml).
• Provide screenshots from your QA / tests locally in the PR description. The goal here is to use the JAR of the plugin and directly test it locally in Kestra UI to ensure it integrates well.

📤 Outputs

• Do not send back as outputs the same infos you already have in your properties.
• If you do not have any output use VoidOutput.
• Do not output twice the same infos (eg: a status code, an error code saying the same thing...).

[fdelbrayelle]

Hello @mady20 👋

I put back the Contributor Checklist in your PR description because some of the points are not respected.

In addition could you add a Trigger and RealTrigger too for the use case as asked in the issue please?

[mady20]

Got it, i will do.

[Malaydewangan09]

Hey @mady20 👋, are there any updates on this?

[mady20]

@Malaydewangan09 I’m sorry, but I won’t be able to continue working on this.