Adding send action for kinesis destination #3397

mdkhan-tw · 2025-11-05T08:20:51Z

Adding send action for kinesis destination
Added aws-kinesis library
Added unit tests

Testing

Added unit tests
Performed end to end testing using action tester, debug logs

arn:aws:iam::000000000000:role/mdkhan-assume-role-for-kinesis-destination mdkhan-external-id
✅ Sent 1 records to Kinesis
Response: {
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'cea4d02d-ecb2-9611-af27-5dd931044082',
    extendedRequestId: 'VEqlAQ6YZHE0mp6OsO+Br2ohbOdzelmuVZsYYeSOox3mm3D2GPW9Nde38lZLkvug0nDzalU9LkALmPC3XJGWmTPyMNGHDto/',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  FailedRecordCount: 0,
  Records: [
    {
      SequenceNumber: '49668576224122157143775420512337700422063236647695155234',
      ShardId: 'shardId-000000000002'
    }
  ]
}

Added unit tests for new functionality
Tested end-to-end using the local server
[If destination is already live] Tested for backward compatibility of destination. Note: New required fields are a breaking change.
[Segmenters] Tested in the staging environment
[Segmenters] [If applicable for this change] Tested for regression with Hadron.

codecov · 2025-11-05T08:27:28Z

Codecov Report

❌ Patch coverage is 93.33333% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.36%. Comparing base (cf48806) to head (9bfa337).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
...actions/src/destinations/aws-kinesis/send/index.ts	50.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3397      +/-   ##
==========================================
+ Coverage   80.33%   80.36%   +0.02%     
==========================================
  Files        1267     1485     +218     
  Lines       25246    28785    +3539     
  Branches     5185     6066     +881     
==========================================
+ Hits        20282    23133    +2851     
- Misses       4176     4844     +668     
- Partials      788      808      +20

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

itsarijitray · 2025-11-05T09:25:51Z

packages/destination-actions/src/destinations/aws-kinesis/utils.ts

+      credentials: credentials
+    })
+
+    await client.send(command)


Can you add the AbortSignal Support similar to s3 destination?
#3381

This will help us standardize timeouts in the system as part of Flexible Request Durations
: https://docs.google.com/document/d/1snhMygiEaMJ2QwC68PjNTmNQdvUhwhqcbTVzlJY4XQA/edit?tab=t.xja93wmhydze

Imo, kinesis destination is very different from S3 or SFTP destinations because in S3/SFTP destinations, we upload very large objects, where the latency will be high and there is a chance that the connection might hang. In Kinesis API call, the max records can't exceed beyond 10 MB / 500 records per call, thus the corresponding latency will be very low (in order of few seconds). I plan to add timeout by connectionTimeout and socketTimeout which is simple to implement that AbortSignal. JIRA https://twilio-engineering.atlassian.net/browse/STRATCONN-6335

Thoughts?

Irrespective of payload sizes, the abortsignal should be implemented as it also handles centrifuge side timeouts/request cancellations, making sure all process are reliably cancelled. However, you can address this in a separate PR as a ticket for there seems to be a separate ticket for this.

Sure, let's take it separately.

Acking that abort signal will be implemented later.

joe-ayoub-segment

Hi Mohammed - I reviewed some of the code and left some comments. But decided to ping you to catch up in person...

packages/destination-actions/src/destinations/aws-kinesis/send/index.ts

packages/destination-actions/src/destinations/aws-kinesis/utils.ts

packages/actions-shared/src/aws-config/index.ts

joe-ayoub-segment · 2025-11-10T12:45:28Z

packages/destination-actions/src/destinations/aws-kinesis/utils.ts

+const validatePartitionKey = (partitionKey: string): void => {
+  if (!partitionKey) {
+    throw new IntegrationError(
+      `Partition Key is required in the payload to send data to Kinesis.`,
+      'PARTITION_KEY_MISSING',
+      400
+    )
+  }
+}


This function shouldn't be required. partition_key is a required field, so only payloads containing a value will be passed into the perform() and performBatch() functions.

imo, it's better to write necessary validations than to only rely on clients. It will be helpful during local testing or traffic from other source than centrifuge (in future)

packages/destination-actions/src/destinations/aws-kinesis/utils.ts

joe-ayoub-segment · 2025-11-10T12:48:18Z

packages/destination-actions/src/destinations/aws-kinesis/utils.ts

+    region: awsRegion,
+    credentials: credentials,
+    requestHandler: new NodeHttpHandler({
+      requestTimeout: KINESIS_COMMAND_TIMEOUT_MS // timeout in milliseconds


Is this the correct timeout value to set?

We will revisit it in the ticket https://twilio-engineering.atlassian.net/browse/STRATCONN-6335

joe-ayoub-segment · 2025-11-10T12:49:18Z

packages/destination-actions/src/destinations/aws-kinesis/utils.ts

+      credentials: credentials
+    })
+
+    await client.send(command)


Acking that abort signal will be implemented later.

joe-ayoub-segment · 2025-11-12T12:45:25Z

PR deployed

viralpickaxe · 2025-11-12T21:18:03Z

Hi @joe-ayoub-segment it appears that this change has broken our entire segment setup as process.env is trying to be read a browser environment

johndellavecchia · 2025-11-12T23:03:56Z

@joe-ayoub-segment this seems to have broken our segment setup. The erroring file is in Destination/actions-shared/dist/aws-config/index.js, which is Segment's internal "Actions / Destinations" bundle.

This reverts commit dc96058.

itsarijitray · 2025-11-13T03:00:42Z

Hello @viralpickaxe @johndellavecchia ,

Thank you for reporting this issue. We sincerely apologize for the disruption this change has caused to your Segment setup.

We are currently prioritizing a revert of the recent change to resolve the process.env issue occurring in the browser environment. We are working on deploying the fix as quickly as possible.

We will provide an update here once the revert is complete and the fix is live.

Thank you for your patience and understanding.

mdkhan-tw added 6 commits November 4, 2025 10:54

merge mz-kinesis

cbda956

refactor

3a9029c

merge main

7e11c63

tests

3c8390f

Merge remote-tracking branch 'origin/main' into STRATCONN-6277

45bc7e5

unit tests

1a51f6d

mdkhan-tw requested a review from a team as a code owner November 5, 2025 08:20

github-actions bot added mode:cloud team:segment-core labels Nov 5, 2025

github-actions bot requested review from itsarijitray and joe-ayoub-segment November 5, 2025 08:21

joe-ayoub-segment added the not-partner-eng-review (cloud) label Nov 5, 2025

itsarijitray reviewed Nov 5, 2025

View reviewed changes

joe-ayoub-segment requested changes Nov 6, 2025

View reviewed changes

joe-ayoub-segment and others added 4 commits November 6, 2025 08:22

Joe meeting with Mohammed

0d9b26d

local changes

3bc1092

resolving PR comments

1223241

adding tests

a5c11c1

github-actions bot added the deploy:push label Nov 7, 2025

removign comments + local code

9bfa337

itsarijitray approved these changes Nov 10, 2025

View reviewed changes

joe-ayoub-segment requested changes Nov 10, 2025

View reviewed changes

joe-ayoub-segment added the Ready for Release label Nov 12, 2025

joe-ayoub-segment approved these changes Nov 12, 2025

View reviewed changes

joe-ayoub-segment merged commit dc96058 into main Nov 12, 2025
22 of 24 checks passed

joe-ayoub-segment deleted the STRATCONN-6277 branch November 12, 2025 10:27

itsarijitray added a commit that referenced this pull request Nov 13, 2025

Revert "Adding send action for kinesis destination (#3397)"

83507f2

This reverts commit dc96058.

itsarijitray mentioned this pull request Nov 13, 2025

Revert "Adding send action for kinesis destination" #3408

Open

Adding send action for kinesis destination #3397

Adding send action for kinesis destination #3397

Uh oh!

Conversation

mdkhan-tw commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

codecov bot commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joe-ayoub-segment left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joe-ayoub-segment commented Nov 12, 2025

Uh oh!

viralpickaxe commented Nov 12, 2025

Uh oh!

johndellavecchia commented Nov 12, 2025

Uh oh!

itsarijitray commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mdkhan-tw commented Nov 5, 2025 •

edited

Loading

codecov bot commented Nov 5, 2025 •

edited

Loading