Upload class

[etcart]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

version 3.896+
Upload class, when given a stream (i.e. from fs.createReadStream) which is larger than the partSize (5MB)
"fails" after uploading

{"level":"info","message":"{\n  loaded: 5242880,\n  total: 0,\n  part: 1,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:23:59.432Z"}
{"level":"info","message":"{\n  loaded: 10485760,\n  total: 0,\n  part: 2,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:23:59.433Z"}
{"level":"info","message":"{\n  loaded: 15728640,\n  total: 0,\n  part: 3,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:23:59.606Z"}
{"level":"info","message":"{\n  loaded: 20971520,\n  total: 0,\n  part: 4,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:23:59.812Z"}
{"level":"info","message":"{\n  loaded: 26214400,\n  total: 0,\n  part: 5,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:23:59.847Z"}
{"level":"info","message":"{\n  loaded: 31457280,\n  total: 0,\n  part: 6,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:23:59.977Z"}
{"level":"info","message":"{\n  loaded: 36700160,\n  total: 0,\n  part: 7,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:24:00.157Z"}
{"level":"info","message":"{\n  loaded: 41943040,\n  total: 0,\n  part: 8,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:24:00.326Z"}
{"level":"info","message":"{\n  loaded: 47185920,\n  total: 0,\n  part: 9,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:24:00.496Z"}
{"level":"info","message":"{\n  loaded: 52428800,\n  total: 0,\n  part: 10,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:24:00.675Z"}
{"level":"info","message":"{\n  loaded: 57671680,\n  total: 0,\n  part: 11,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:24:00.907Z"}
{"level":"info","message":"{\n  loaded: 62914560,\n  total: 0,\n  part: 12,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:24:01.130Z"}
{"level":"info","message":"{\n  loaded: 68157440,\n  total: 0,\n  part: 13,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:24:01.322Z"}
{"level":"info","message":"{\n  loaded: 69206016,\n  total: 0,\n  part: 16,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:24:02.802Z"}
{"level":"info","message":"{\n  loaded: 74448896,\n  total: 0,\n  part: 14,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:24:03.271Z"}
{"level":"info","message":"{\n  loaded: 79691776,\n  total: 0,\n  part: 15,\n  Key: 'source-key-e89738',\n  Bucket: 'source-bucket-4e310c'\n}","sender":"aws-client/s3","timestamp":"2025-09-26T23:24:03.290Z"}

...<test output>...

  Error {
    message: 'Expected 0 part(s) but uploaded 16 part(s).',
  }

this appears to successfully upload, but then fails on validation.
a check of values in the constructor shows that

this.totalBytes = byteLength(this.params.Body) // 0 (lib-storage/src/Upload.ts LN 92)
this.partSize = Math.max(Upload.MIN_PART_SIZE, Math.floor((this.totalBytes || 0) / this.MAX_PARTS)); // 0 (lib-storage/src/Upload.ts LN 96)

checking version < 3.895 and 3.894, the functionality operates without throwing an error, but a check of the value in "this.totalBytes" shows that is also 0, there's just no validation happening to "cause problems" because of this value. it does in any of the tested versions throwing errors or not, cause the total value in that above log example, to be 0

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/package-name@3.896+

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.12.2

Reproduction Steps

const { Upload } = require('@aws-sdk/lib-storage');
const {
  S3Client,
  CreateBucketCommand,
  DeleteBucketCommand,
  DeleteObjectCommand,
  waitUntilObjectNotExists
} = require("@aws-sdk/client-s3");
const { v4: uuidv4 } = require('uuid');
const fs = require('fs');

const Bucket = 'deleteme' + uuidv4();

const MB = 1024*1024;
const size = 6*MB // (or larger)
const readStream = fs.createReadStream('/dev/urandom', { end: size - 1 });
const s3 = new S3Client({});

await s3.send(new CreateBucketCommand({Bucket}));

const parallelUploads = new Upload({
  params: {
    Bucket,
    Key: "deleteme",
    Body: readStream,
  },
  client: s3
});


const result = await parallelUploads.done() //this should actually throw on 3.896+

await s3.send(new DeleteObjectCommand({ Bucket, Key: "deleteme" }))
await s3.send(new DeleteBucketCommand({ Bucket }))

Observed Behavior

Uncaught Error: Expected 0 part(s) but uploaded 2 part(s).
    at _Upload.__doMultipartUpload (/Users/ecarton/cumulus/packages/aws-client/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:434:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async _Upload.done (/Users/ecarton/cumulus/packages/aws-client/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:235:12)

Expected Behavior

upload completes without issue

Possible Solution

No response

Additional Information/Context

No response

[trivikr]

I can verify that the issue exists

Tested with code

import { Upload } from "@aws-sdk/lib-storage";
import { S3 } from "@aws-sdk/client-s3";
import { createReadStream } from "fs";

const Bucket = "test-aws-sdk-js-v3-" + Math.random();

const MB = 1024 * 1024;
const size = 6 * MB; // (or larger)
const readStream = createReadStream("/dev/urandom", { end: size - 1 });

const s3 = new S3();

await s3.createBucket({ Bucket });

const parallelUploads = new Upload({
  params: { Bucket, Key: "deleteme", Body: readStream },
  client: s3,
});

await parallelUploads.done(); //this should actually throw on 3.896+

await s3.deleteObject({ Bucket, Key: "deleteme" });
await s3.deleteBucket({ Bucket });

The upload succeeds for v3.895.0, but fails for v3.896.0

[trivikr]

We've npm deprecated the affected version https://www.npmjs.com/package/@aws-sdk/lib-storage/v/3.896.0

While we work on a fix, please pin @aws-sdk/lib-storage version to <=v3.895.0 in your package.json

[trivikr]

Lowering to p2, as this issue is specific to /dev/random for which Node.js doesn't return file size.
Source code: https://github.com/smithy-lang/smithy-typescript/blob/58c7732ccb3d37a38d238019f38c7e1c136eeedf/packages/util-body-length-node/src/calculateBodyLength.ts#L19-L22

The issue is not reproducible if we create a file as seen in minimal repro below

import { Upload } from "@aws-sdk/lib-storage";
import { S3 } from "@aws-sdk/client-s3";
import { createReadStream, writeFileSync } from "fs";

const Bucket = "test-aws-sdk-js-v3-" + Math.random();

// Generate 6MB of random data
const MB = 1024 * 1024;
const size = 6 * MB;
const FILE_NAME = "random-6mb.dat";

writeFileSync(FILE_NAME, Buffer.alloc(size));
const readStream = createReadStream(FILE_NAME);

const s3 = new S3();

await s3.createBucket({ Bucket });

const parallelUploads = new Upload({
  params: { Bucket, Key: "deleteme", Body: readStream },
  client: s3,
});

await parallelUploads.done();

await s3.deleteObject({ Bucket, Key: "deleteme" });
await s3.deleteBucket({ Bucket });

And the implementation in #7363 performs the validation for content length correctly as required.

[trivikr]

I've undeprecated the version on npm
https://www.npmjs.com/package/@aws-sdk/lib-storage/v/3.896.0

@etcart Are you facing this issue with files streams with fixed size, like the one shown in the minimal repro in previous comment? Or are you seeing this error with some other streams?

[etcart]

thank you for the follow up, I've tested with this configuration and it works fine. I agree with the de-prioritization that makes this a fairly niche issue. we're using the above system for testing and do want this nice low memory footprint, low disk footprint system, but we can work around with this.

[kuhe]

We intend to, in a future version, make the calculation of the total part size take into account the user-supplied ContentLength property (optional).

This allows the caller to supply the byte length for irregular cases like /dev/urandom where the file size reported does not match the true byte length of the stream.