Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(middleware-flexible-checksums): use Node.js native CRC32 checksum API #6641

Merged
merged 3 commits into from
Nov 8, 2024

Conversation

trivikr
Copy link
Member

@trivikr trivikr commented Nov 7, 2024

Issue

Description

Uses Node.js native CRC32 checksum API, when available

Testing

import assert from "assert";
import { S3 } from "../aws-sdk-js-v3/clients/client-s3/dist-cjs/index.js";

const s3 = new S3({ region: "us-west-2" });
const Bucket = "test-flexible-checksums"; // Change to your bucket name
const Key = "test.txt";
const Body = "helloworld".repeat(10_000_000); // A string of size ~100MB

const putObjectStart = performance.now();
await s3.putObject({ Bucket, Key, Body, ChecksumAlgorithm: "CRC32" });
console.log("time taken for putObject request:", (performance.now() - putObjectStart).toFixed(2), "ms");

const getObjectStart = performance.now();
const response = await s3.getObject({ Bucket, Key, ChecksumMode: "ENABLED" });
console.log("time taken for getObject request:", (performance.now() - getObjectStart).toFixed(2), "ms");

const transformStart = performance.now();
const body = await response.Body.transformToString();
console.log("time taken for transform (checksum validation):", (performance.now() - transformStart).toFixed(2), "ms");

assert(body === Body);

Without Node.js native CRC32 (node v20.10.0)

time taken for putObject request: 3399.14 ms
time taken for getObject request: 1145.27 ms
time taken for transform (checksum validation): 2137.78 ms

With Node.js native CRC32 (node v20.17.0)

time taken for putObject request: 1427.59 ms
time taken for getObject request: 1141.85 ms
time taken for transform (checksum validation): 246.76 ms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@trivikr trivikr force-pushed the checksum-node-native branch from 238a9d5 to 2cd531b Compare November 7, 2024 23:34
@trivikr trivikr marked this pull request as ready for review November 7, 2024 23:54
@trivikr trivikr requested a review from a team as a code owner November 7, 2024 23:54
@trivikr

This comment was marked as outdated.

@trivikr trivikr merged commit 6a48760 into aws:main Nov 8, 2024
4 checks passed
@trivikr trivikr deleted the checksum-node-native branch November 8, 2024 16:21
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 23, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants