feat(aws.emrserverless): add EMR Serverless plugin (create, submit, delete)

[aazam-gh]

What changes are being made and why?

---

This PR introduces AWS EMR Serverless support to the plugin-aws repository.
It extends Kestra’s existing EMR integration beyond cluster-based execution, enabling users to build, run, and manage EMR Serverless workflows natively in Kestra.
Closes #579

How the changes have been QAed?

---

Setup Instructions

---

Contributor Checklist ✅

• PR Title and commits follows conventional commits
• Add a closes #ISSUE_ID or fixes #ISSUE_ID in the description if the PR relates to an opened issue.
• Documentation updated (plugin docs from @Schema for properties and outputs, @Plugin with examples, README.md file with basic knowledge and specifics).
• Setup instructions included if needed (API keys, accounts, etc.).
• Prefix all rendered properties by r not rendered (eg: rHost).
• Use runContext.logger() to log enough important infos where it's needed and with the best level (DEBUG, INFO, WARN or ERROR).

⚙️ Properties

• Properties are declared with Property<T> carrier type, do not use @PluginProperty.
• Mandatory properties must be annotated with @NotNull and checked during the rendering.
• You can model a JSON thanks to a simple Property<Map<String, Object>>.

🌐 HTTP

• Must use Kestra’s internal HTTP client from io.kestra.core.http.client

📦 JSON

• If you are serializing response from an external API, you may have to add a @JsonIgnoreProperties(ignoreUnknown = true) at the mapped class level. So that we will avoid to crash the plugin if the provider add a new field suddenly.
• Must use Jackson mappers provided by core (io.kestra.core.serializers)

✨ New plugins / subplugins

• Make sure your new plugin is configured like mentioned here.
• Add a package-info.java under each sub package respecting this format and choosing the right category.
• Icons added in src/main/resources/icons in SVG format and not in thumbnail (keep it big):
  • plugin-icon.svg
  • One icon per package, e.g. io.kestra.plugin.aws.svg
  • For subpackages, e.g. io.kestra.plugin.aws.s3, add io.kestra.plugin.aws.s3.svg
    See example here.
• Use "{{ secret('YOUR_SECRET') }}" in the examples for sensible infos such as an API KEY.
• If you are fetching data (one, many or too many), you must add a Property<FetchType> fetchType to be able to use FETCH_ONE, FETCH and even STORE to store big amount of data in the internal storage.
• Align the """ to close examples blocks with the flow id.

🧪 Tests

• Unit Tests added or updated to cover the change (using the RunContext to actually run tasks).
• Add sanity checks if possible with a YAML flow inside src/test/resources/flows.
• Avoid disabling tests for CI. Instead, configure a local environment whenever it's possible with .github/setup-unit.sh (which can be executed locally and in the CI) all along with a new docker-compose-ci.yml file (do not edit the existing docker-compose.yml).
• Provide screenshots from your QA / tests locally in the PR description. The goal here is to use the JAR of the plugin and directly test it locally in Kestra UI to ensure it integrates well.

📤 Outputs

• Do not send back as outputs the same infos you already have in your properties.
• If you do not have any output use VoidOutput.
• Do not output twice the same infos (eg: a status code, an error code saying the same thing...).

[aazam-gh]

This PR is currently paused in order to obtain an increased EMR serverless quota for instances for testing.

[fdelbrayelle]

Any news on this @aazam-gh ?

[aazam-gh]

Followed up with AWS support team again. Are there any alternatives for testing?

[Malaydewangan09]

Hey @aazam-gh 👋, did you try using localStack for testing?

[aazam-gh]

@Malaydewangan09 Not yet, I will try that now. thanks!

[MilosPaunovic]

Hey @aazam-gh, are there any updates on this?

[aazam-gh]

Apparently localstack does not support emr serverless application features yet.
And i received an updated email for the AWS support team that my service quota is still under process ):

[Malaydewangan09]

@aazam-gh, could you please at least write some unit tests for these new tasks? So that we can check on our side too, Thanks!

[fdelbrayelle]

Any news from the AWS Support for your quota @aazam-gh ?

[fdelbrayelle]

QA

Create and Start a Serverless Application

Flow:

id: create_and_run_emr_serverless
namespace: company.team

tasks:
  - id: create_and_run
    type: io.kestra.plugin.aws.emr.CreateServerlessApplicationAndStartJob
    accessKeyId: "${{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "${{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-north-1"
    releaseLabel: "emr-7.12.0"
    applicationType: "SPARK"
    executionRoleArn: "arn:aws:iam::707969873520:role/EMRServerlessRole"
    jobName: "example-job"
    entryPoint: "s3://my-bucket/jobs/script.py"

Execution Gantt result:

The app is created and visible in EMR Studio:

Delete a Serverless Application

Here are the 2 existing apps we will delete:

Flow:

id: aws_emrserverless_delete_app
namespace: company.team

tasks:
  - id: delete_app
    type: io.kestra.plugin.aws.emr.DeleteServerlessApplication
    accessKeyId: "${{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "${{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-north-1"
    applicationIds:
        - 00g1dvc093m5tj1d
        - 00g1dvbn2429t01d

Execution Gantt result:

The 2 flows were deleted from EMR Studio:

Start an existing Serverless Application

Let's take this stopped app:

Flow:

id: start_emr_job
namespace: company.team

tasks:
  - id: start_job
    type: io.kestra.plugin.aws.emr.StartServerlessJobRun
    accessKeyId: "${{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "${{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-north-1"
    applicationId: "00g1dvclhbhl781d"
    executionRoleArn: "arn:aws:iam::707969873520:role/EMRServerlessRole"
    jobName: "example-job"
    entryPoint: "s3://my-bucket/scripts/spark-app.py"

Execution Gantt result:

The application appears now has "started" in EMR Studio:

[fdelbrayelle]

Hello @aazam-gh 👋 I added a bunch of unit tests and did a QA which passed ✔️ So congratulations and thank you! I'll merge and do a release!