Example Next.js app using duckdb-wasm to read/fetch Parquet files, in Node and the browser.
Demo: runsascoded.com/next-duckdb-parquet-demo
- pages/index.tsx: shows 3 ways of loading Parquet data (on a mix of client and server):
useParquetBuf
: load parquet file as byte array on server, send to client, decode in browser (minimize data sent to client while loading a static page)loadparquet
: load parquet file on server, send to clientuseParquet
: fetch parquet file by URL on client
- src/parquet.ts: utilities for instantiating DuckDB and fetching/loading Parquet data
- Copied from runsascoded/next-utils/parquet.ts, adapted from ilyabo/graphnavi/lib/useDuckConn.ts
- See also duckdb-wasm#1148
- public/people.parquet: demo Parquet file, generated by init-data.js
runsascoded.com/next-duckdb-parquet-demo:
Parquet data loaded on server (in
getStaticProps
, withloadParquet
):[{"id":1,"name":"Mark"},{"id":2,"name":"Hannes"}]Parquet data buffer loaded on server, parsed on client (with
useParquetBuf
):[{"id":1,"name":"Mark"},{"id":2,"name":"Hannes"}]Parquet data fetched on client, from https://runsascoded.com/next-duckdb-parquet-demo/people.parquet (with
useParquet
):[{"id":1,"name":"Mark"},{"id":2,"name":"Hannes"}]
Dev mode:
next dev # serve dev-mode site at http://127.0.0.1:3000
Build + Export:
next build
next export
http-server out # serve static site from out/ dir
node init-data.js
Runs init-data.js:
const duckdb = require('duckdb')
const db = new duckdb.Database(':memory:')
db.run(`
CREATE TABLE people(id INTEGER, name VARCHAR);
INSERT INTO people VALUES (1, 'Mark'), (2, 'Hannes');
COPY (SELECT * FROM people) TO 'public/people.parquet' (FORMAT 'parquet');
`)
For some reason, the server-side loadParquet
path doesn't seem to work under Next.js's new "App router" (also reported at next.js#57819).
Here's a trimmed down example (from the @app-broken branch):
- pages/pages.tsx ✅ works
- app/app/page.tsx ❌ broken
Attempting to load the latter emits this on the server side:
duckdb-wasm fetch
bestBundle: {
mainModule: '/Users/ryan/c/next-duckdb-parquet-demo/node_modules/@duckdb/duckdb-wasm/dist/duckdb-eh.wasm',
mainWorker: '/Users/ryan/c/next-duckdb-parquet-demo/node_modules/@duckdb/duckdb-wasm/dist/duckdb-node-eh.worker.cjs',
pthreadWorker: null
}
made logger
made db
worker terminated with 1 pending requests
and the client hangs.
Similarly, next build
times out, meaning I can't deploy a demo. Here's a GitHub Actions run that failed due to this issue.
diff pages/pages.tsx app/app/page.tsx
:
-export async function getStaticProps() {
- const data = await loadParquet<Person>(parquetPath) // ✅ works fine under pages/
- return { props: { data } }
-}
-
-export default function Home({ data }: { data: Person[] }) {
+export default async function Home() {
+ const data = await loadParquet<Person>(parquetPath) // ❌ broken under app/
await db.instantiate(bundle.mainModule, bundle.pthreadWorker);
emits this error:
worker terminated with 1 pending requests
I haven't found any further leads about what might be going on.