duckdb-nodejs-layer

Packaging DuckDB for usage in AWS Lambda functions with Node.js, and publishing as public Lambda layers.

Usage

You can use the published layers in your own serverless applications by referencing it as outlined in the different framework's docs:

The layers get automatically published to all currently available (in the moment of publishing) AWS regions.

Example usage

You can have a look at the example repository, which uses this AWS Lambda layer: tobilg/serverless-duckdb. More specifically, the src/functions/query.js should give you a good idea on how it can be used.

The recommendation is to use a bundler such as WebPack for the packaging of your application. This means you can locally use (e.g. for testing) the official duckdb npm package (via npm i --save duckdb), but exclude it in the packaging process, because after being deployed to a Lambda function, the updated DuckDB version from the Lambda layer will be used.

Code example

import DuckDB from 'duckdb';

// Instantiate DuckDB
const duckDB = new DuckDB.Database(':memory:');

// Create connection
const connection = duckDB.connect();

// Promisify query method
const query = (query) => {
  return new Promise((resolve, reject) => {
    connection.all(query, (err, res) => {
      if (err) reject(err);
      resolve(res);
    })
  })
}

// Will show DuckDB version
await query(`PRAGMA version;`);

DuckDB layer

This contains a AWS Lambda layer in different flavors, both x86 and arm64 architectures are supported.

The ARNs follow the following logic:

arn:aws:lambda:$REGION:041475135427:layer:duckdb-nodejs-$ARCHITECTURE:$VERSION

where $ARCHITECTURE can have the following values:

x86
arm64

Enabled extensions

The following DuckDB default extensions are enabled and contained in the static build:

parquet: Adds support for reading and writing parquet files
httpfs: Adds support for reading and writing files over a HTTP(S) connection
json: Adds support for JSON operations
fts: Adds support for Full-Text Search Indexes
icu: Adds support for time zones and collations using the ICU library

x86 layer ARNs

Layer version to DuckDB version mapping:

Layer version	DuckDB version
3	v0.8.0
4	v0.8.1
5	v0.9.0
6	v0.9.1
7	v0.9.2
8	v0.10.0
9	v0.10.1
10	v0.10.2
11	v1.0.0
12	v1.1.0
13	v1.1.1
14	v1.1.3

The ARNs of the latest x86 version of the DuckDB Node.js Lambda layer are:

Region	Layer ARN
af-south-1	arn:aws:lambda:af-south-1:041475135427:layer:duckdb-nodejs-x86:14
ap-east-1	arn:aws:lambda:ap-east-1:041475135427:layer:duckdb-nodejs-x86:14
ap-northeast-1	arn:aws:lambda:ap-northeast-1:041475135427:layer:duckdb-nodejs-x86:14
ap-northeast-2	arn:aws:lambda:ap-northeast-2:041475135427:layer:duckdb-nodejs-x86:14
ap-northeast-3	arn:aws:lambda:ap-northeast-3:041475135427:layer:duckdb-nodejs-x86:14
ap-south-1	arn:aws:lambda:ap-south-1:041475135427:layer:duckdb-nodejs-x86:14
ap-south-2	arn:aws:lambda:ap-south-2:041475135427:layer:duckdb-nodejs-x86:14
ap-southeast-1	arn:aws:lambda:ap-southeast-1:041475135427:layer:duckdb-nodejs-x86:14
ap-southeast-2	arn:aws:lambda:ap-southeast-2:041475135427:layer:duckdb-nodejs-x86:14
ap-southeast-3	arn:aws:lambda:ap-southeast-3:041475135427:layer:duckdb-nodejs-x86:14
ap-southeast-4	arn:aws:lambda:ap-southeast-4:041475135427:layer:duckdb-nodejs-x86:14
ca-central-1	arn:aws:lambda:ca-central-1:041475135427:layer:duckdb-nodejs-x86:14
eu-central-1	arn:aws:lambda:eu-central-1:041475135427:layer:duckdb-nodejs-x86:14
eu-central-2	arn:aws:lambda:eu-central-2:041475135427:layer:duckdb-nodejs-x86:14
eu-north-1	arn:aws:lambda:eu-north-1:041475135427:layer:duckdb-nodejs-x86:14
eu-south-1	arn:aws:lambda:eu-south-1:041475135427:layer:duckdb-nodejs-x86:14
eu-south-2	arn:aws:lambda:eu-south-2:041475135427:layer:duckdb-nodejs-x86:14
eu-west-1	arn:aws:lambda:eu-west-1:041475135427:layer:duckdb-nodejs-x86:14
eu-west-2	arn:aws:lambda:eu-west-2:041475135427:layer:duckdb-nodejs-x86:14
eu-west-3	arn:aws:lambda:eu-west-3:041475135427:layer:duckdb-nodejs-x86:14
me-central-1	arn:aws:lambda:me-central-1:041475135427:layer:duckdb-nodejs-x86:14
me-south-1	arn:aws:lambda:me-south-1:041475135427:layer:duckdb-nodejs-x86:14
sa-east-1	arn:aws:lambda:sa-east-1:041475135427:layer:duckdb-nodejs-x86:14
us-east-1	arn:aws:lambda:us-east-1:041475135427:layer:duckdb-nodejs-x86:14
us-east-2	arn:aws:lambda:us-east-2:041475135427:layer:duckdb-nodejs-x86:14
us-west-1	arn:aws:lambda:us-west-1:041475135427:layer:duckdb-nodejs-x86:14
us-west-2	arn:aws:lambda:us-west-2:041475135427:layer:duckdb-nodejs-x86:14

arm64 layer ARNs

Layer version to DuckDB version mapping:

Layer version	DuckDB version
1	v0.8.0
2	v0.8.1
3	v0.9.0
4	v0.9.1
5	v0.9.2
6	v0.10.0
7	v0.10.1
8	v0.10.2
9	v1.0.0
10	v1.1.0
11	v1.1.1
12	v1.1.3

The ARNs of the latest arm64 version of the DuckDB Node.js Lambda layer are:

Region	Layer ARN
us-east-1	arn:aws:lambda:us-east-1:041475135427:layer:duckdb-nodejs-arm64:12
us-east-2	arn:aws:lambda:us-east-2:041475135427:layer:duckdb-nodejs-arm64:12
us-west-1	arn:aws:lambda:us-west-1:041475135427:layer:duckdb-nodejs-arm64:12
us-west-2	arn:aws:lambda:us-west-2:041475135427:layer:duckdb-nodejs-arm64:12
af-south-1	arn:aws:lambda:af-south-1:041475135427:layer:duckdb-nodejs-arm64:12
ap-east-1	arn:aws:lambda:ap-east-1:041475135427:layer:duckdb-nodejs-arm64:12
ap-southeast-3	arn:aws:lambda:ap-southeast-3:041475135427:layer:duckdb-nodejs-arm64:12
ap-south-1	arn:aws:lambda:ap-south-1:041475135427:layer:duckdb-nodejs-arm64:12
ap-northeast-3	arn:aws:lambda:ap-northeast-3:041475135427:layer:duckdb-nodejs-arm64:12
ap-northeast-2	arn:aws:lambda:ap-northeast-2:041475135427:layer:duckdb-nodejs-arm64:12
ap-southeast-1	arn:aws:lambda:ap-southeast-1:041475135427:layer:duckdb-nodejs-arm64:12
ap-southeast-2	arn:aws:lambda:ap-southeast-2:041475135427:layer:duckdb-nodejs-arm64:12
ap-northeast-1	arn:aws:lambda:ap-northeast-1:041475135427:layer:duckdb-nodejs-arm64:12
ca-central-1	arn:aws:lambda:ca-central-1:041475135427:layer:duckdb-nodejs-arm64:12
eu-central-1	arn:aws:lambda:eu-central-1:041475135427:layer:duckdb-nodejs-arm64:12
eu-west-1	arn:aws:lambda:eu-west-1:041475135427:layer:duckdb-nodejs-arm64:12
eu-west-2	arn:aws:lambda:eu-west-2:041475135427:layer:duckdb-nodejs-arm64:12
eu-south-1	arn:aws:lambda:eu-south-1:041475135427:layer:duckdb-nodejs-arm64:12
eu-west-3	arn:aws:lambda:eu-west-3:041475135427:layer:duckdb-nodejs-arm64:12
eu-north-1	arn:aws:lambda:eu-north-1:041475135427:layer:duckdb-nodejs-arm64:12
me-south-1	arn:aws:lambda:me-south-1:041475135427:layer:duckdb-nodejs-arm64:12
sa-east-1	arn:aws:lambda:sa-east-1:041475135427:layer:duckdb-nodejs-arm64:12

x86_64 layer with arrow, aws and httpfs extensions ARNs

This layer contains the arrow, aws and httpfs extensions. This enables Lambda functions that are in a VPC and have no internet access to download these extensions via HTTP (e.g. from extensions.quacking.cloud) for cost savings.

VPC access is for example necessary if you want your Lambda functions to access S3 Express One Zone. You can have a look at how to deploy such kind of setup in the tobilg/serverless-duckdb repo.

Currently, it's only available in us-east-1.

Layer version to DuckDB version mapping:

Layer version	DuckDB version
1	v0.10.0
2	v0.10.1
4	v0.10.2
5	v1.0.0
6	v1.1.0

The ARNs of the latest x86_64 version of the DuckDB Node.js Lambda layer with extensions are:

Region	Layer ARN
us-east-1	arn:aws:lambda:us-east-1:041475135427:layer:duckdb-extensions-nodejs-x86:6

Name		Name	Last commit message	Last commit date
Latest commit History 199 Commits
.github/workflows		.github/workflows
release		release
src		src
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile.arm64		Dockerfile.arm64
Dockerfile.extensions-all.x86_64		Dockerfile.extensions-all.x86_64
Dockerfile.extensions-base.x86_64		Dockerfile.extensions-base.x86_64
Dockerfile.with-extensions.x86_64		Dockerfile.with-extensions.x86_64
Dockerfile.x86_64		Dockerfile.x86_64
LICENSE		LICENSE
README.md		README.md
publish_extensions_layer.sh		publish_extensions_layer.sh
publish_layer.sh		publish_layer.sh
publish_package.sh		publish_package.sh

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

duckdb-nodejs-layer

Usage

Example usage

Code example

DuckDB layer

Enabled extensions

x86 layer ARNs

arm64 layer ARNs

x86_64 layer with arrow, aws and httpfs extensions ARNs

About

Releases

Languages

License

tobilg/duckdb-nodejs-layer

Folders and files

Latest commit

History

Repository files navigation

duckdb-nodejs-layer

Usage

Example usage

Code example

DuckDB layer

Enabled extensions

x86 layer ARNs

arm64 layer ARNs

x86_64 layer with arrow, aws and httpfs extensions ARNs

About

Resources

License

Stars

Watchers

Forks

Releases

Languages