Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: database migrations #333

Merged
merged 30 commits into from
Oct 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
fb6be5e
feat: include database migrations in build
RihanArfan Oct 21, 2024
326020d
feat: apply migrations in pages ci
RihanArfan Oct 23, 2024
0f0cfa1
Merge branch 'nuxt-hub:main' into feat/migrations
RihanArfan Oct 23, 2024
3c59920
fix: remove calling hooks in non-runtime
RihanArfan Oct 23, 2024
8f22d22
feat: apply migrations in pages ci
RihanArfan Oct 23, 2024
37c1125
refactor: enable usage during dev
RihanArfan Oct 23, 2024
2bfcaae
feat: migrations hook
RihanArfan Oct 23, 2024
ed22ddd
refactor: improve code consistency
RihanArfan Oct 24, 2024
350476a
feat(playground): apply migrations via nuxt hub
RihanArfan Oct 24, 2024
101935b
fix: correctly resolve imports in runtime
RihanArfan Oct 24, 2024
2f7215b
feat: local migrations
RihanArfan Oct 24, 2024
04e78b5
refactor: separate usage of @nuxt/kit & disable migrations during dev…
RihanArfan Oct 24, 2024
9d62b1a
feat: local development migrations
RihanArfan Oct 24, 2024
deec85c
refactor: consistent naming
RihanArfan Oct 24, 2024
537f577
feat: support remote migrations during dev --remote
RihanArfan Oct 24, 2024
1228ad5
feat: log remote env name for self host
RihanArfan Oct 24, 2024
c5819d2
refactor: move logic into runtime
RihanArfan Oct 24, 2024
c9c2377
fix: correctly provide token
RihanArfan Oct 24, 2024
3af9654
chore: message consistency and fix type
RihanArfan Oct 24, 2024
aa3d09f
Merge remote-tracking branch 'upstream/main' into feat/migrations
RihanArfan Oct 24, 2024
6ed8589
docs: migrations docs
RihanArfan Oct 25, 2024
469c005
chore: rename table to _hub_migrations
atinux Oct 25, 2024
d6f7a29
chore: rename hook to database:migrations:done
atinux Oct 25, 2024
50c6052
chore: update
atinux Oct 25, 2024
f2100c2
Merge branch 'main' into feat/migrations
atinux Oct 25, 2024
e0ce88c
docs: update
atinux Oct 25, 2024
d02d1e9
typo
atinux Oct 25, 2024
398851d
chore: og image
atinux Oct 25, 2024
80f8b90
up
atinux Oct 25, 2024
157b1dd
up
atinux Oct 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/content/0.index.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ hero:
light: '/images/landing/hero-light.svg'
dark: '/images/landing/hero-dark.svg'
headline:
label: "Blob: Presigned URLs"
to: /changelog/blob-presigned-urls
label: "Automatic Database Migrations"
to: /changelog/database-migrations
icon: i-ph-arrow-right
features:
- name: Cloud Hosting
Expand Down
131 changes: 117 additions & 14 deletions docs/content/1.docs/2.features/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,20 @@ console.log(results)
*/
```

The method return an object that contains the results (if applicable), the success status and a meta object:

```ts
{
results: array | null, // [] if empty, or null if it does not apply
success: boolean, // true if the operation was successful, false otherwise
meta: {
duration: number, // duration of the operation in milliseconds
rows_read: number, // the number of rows read (scanned) by this query
rows_written: number // the number of rows written by this query
}
}
```

### `first()`

Returns the first row of the results. This does not return metadata like the other methods. Instead, it returns the object directly.
Expand Down Expand Up @@ -202,6 +216,8 @@ console.log(info1)
*/
```

The object returned is the same as the [`.all()`](#all) method.

### `exec()`

Executes one or more queries directly without prepared statements or parameters binding. The input can be one or multiple queries separated by \n.
Expand All @@ -223,22 +239,109 @@ console.log(result)
This method can have poorer performance (prepared statements can be reused in some cases) and, more importantly, is less safe. Only use this method for maintenance and one-shot tasks (for example, migration jobs). The input can be one or multiple queries separated by \n.
::

## Return Object
## Database Migrations

The methods [`.all()`](#all) and [`.batch()`](#batch) return an object that contains the results (if applicable), the success status and a meta object:
Database migrations provide version control for your database schema. They track changes and ensure consistent schema evolution across all environments through incremental updates.

```ts
{
results: array | null, // [] if empty, or null if it does not apply
success: boolean, // true if the operation was successful, false otherwise
meta: {
duration: number, // duration of the operation in milliseconds
rows_read: number, // the number of rows read (scanned) by this query
rows_written: number // the number of rows written by this query
}
}
### Automatic Application

SQL migrations in `server/database/migrations/*.sql` now automatically apply when you:
- Start the development server (`npx nuxt dev` or [`npx nuxt dev --remote`](/docs/getting-started/remote-storage))
- Preview builds locally ([`npx nuxthub preview`](/changelog/nuxthub-preview))
- Deploy via [`npx nuxthub deploy`](/docs/getting-started/deploy#nuxthub-cli) or [Cloudflare Pages CI](/docs/getting-started/deploy#cloudflare-pages-ci)

::tip
All applied migrations are tracked in the `_hub_migrations` database table.
::

### Creating Migrations

Generate a new migration file using:

```bash [Terminal]
npx nuxthub database migrations create <name>
```

::callout
Read more on [Cloudflare D1 documentation](https://developers.cloudflare.com/d1/build-databases/query-databases/).
::important
Migration names must only contain alphanumeric characters and `-` (spaces are converted to `-`).
::

Migration files are created in `server/database/migrations/`.

```bash [Example]
> npx nuxthub database migrations create create-todos
✔ Created ./server/database/migrations/0001_create-todos.sql
```

After creation, add your SQL queries to modify the database schema.


::note{to="/docs/recipes/drizzle#npm-run-dbgenerate"}
With [Drizzle ORM](/docs/recipes/drizzle), migrations are automatically created when you run `npx drizzle-kit generate`.
::

### Checking Migration Status

View pending and applied migrations across environments:

```bash [Terminal]
# Local environment status
npx nuxthub database migrations list

# Preview environment status
npx nuxthub database migrations list --preview

# Production environment status
npx nuxthub database migrations list --production
```

```bash [Example output]
> npx nuxthub database migrations list --production
ℹ Connected to project atidone.
ℹ Using https://todos.nuxt.dev to retrieve migrations.
✔ Found 1 migration on atidone...
✅ ./server/database/migrations/0001_create-todos.sql 10/25/2024, 2:43:32 PM
🕒 ./server/database/migrations/0002_create-users.sql Pending
```

### Marking Migrations as Applied

For databases with existing migrations, prevent NuxtHub from rerunning them by marking them as applied:

```bash [Terminal]
# Mark applied in local environment
npx nuxthub database migrations mark-all-applied

# Mark applied in preview environment
npx nuxthub database migrations mark-all-applied --preview

# Mark applied in production environment
npx nuxthub database migrations mark-all-applied --production
```

::collapsible{name="self-hosting docs"}
When [self-hosting](/docs/getting-started/deploy#self-hosted), set these environment variables before running commands: :br :br

```bash [Terminal]
NUXT_HUB_PROJECT_URL=<url> NUXT_HUB_PROJECT_SECRET_KEY=<secret> nuxthub database migrations mark-all-applied
```
::

### Migrating from Drizzle ORM

Since NuxtHub doesn't recognize previously applied Drizzle ORM migrations (stored in `__drizzle_migrations`), it will attempt to rerun all migrations in `server/database/migrations/*.sql`. To prevent this:

1. Mark existing migrations as applied in each environment:

```bash [Terminal]
# Local environment
npx nuxthub database migrations mark-all-applied

# Preview environment
npx nuxthub database migrations mark-all-applied --preview

# Production environment
npx nuxthub database migrations mark-all-applied --production
```

2. Remove `server/plugins/database.ts` as it's no longer needed.
26 changes: 12 additions & 14 deletions docs/content/1.docs/3.recipes/1.hooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,18 @@ description: Use lifecycle hooks to stay synced with NuxtHub.

## `onHubReady()`

Use `onHubReady()` to ensure the execution of some code once NuxtHub environment bindings are set up.
Use `onHubReady()` to ensure the execution of some code once NuxtHub environment bindings are set up and database migrations are applied.

::note
`onHubReady()` is a shortcut using the [`hubHooks`](#hubhooks) object under the hood to listen to the `bindings:ready` event.
`onHubReady()` is a shortcut using the [`hubHooks`](#hubhooks) object under the hood that listens to the `bindings:ready` and `database:migrations:done` events.
::

This is useful to run database migrations inside your [server/plugins/](https://nuxt.com/docs/guide/directory-structure/server#server-plugins).

```ts [server/plugins/migrations.ts]
export default defineNitroPlugin(() => {
// Only run migrations in development
// Only run in development
if (import.meta.dev) {
onHubReady(async () => {
await hubDatabase().exec(`
CREATE TABLE IF NOT EXISTS todos (
id INTEGER PRIMARY KEY,
title TEXT NOT NULL,
completed INTEGER NOT NULL DEFAULT 0
)
`.replace(/\n/g, ''))
console.log('NuxtHub is ready! 🚀')
})
}
})
Expand All @@ -40,23 +32,29 @@ The `hubHooks` object is a collection of hooks that can be used to stay synced w
```ts [Signature]
export interface HubHooks {
'bindings:ready': () => any | void
'database:migrations:done': () => any | void
}
```

### Usage

You can use the `hubHooks` object to listen to the `bindings:ready` event in your server plugins:
You can use the `hubHooks` object to listen to `HubHooks` events in your server plugins:

```ts [server/plugins/hub.ts]
export default defineNitroPlugin(() => {
hubHooks.hook('bindings:ready', () => {
console.log('NuxtHub bindings are ready!')
const db = hubDatabase()
})
// Only run in development and if the database is enabled
if (import.meta.dev) {
hubHooks.hook('database:migrations:done', () => {
console.log('Database migrations are done!')
})
}
})
```

::note
Note that `hubHooks` is a [hookable](https://hookable.unjs.io) instance.
::

31 changes: 3 additions & 28 deletions docs/content/1.docs/3.recipes/2.drizzle.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,35 +92,10 @@ When running the `npm run db:generate` command, `drizzle-kit` will generate the

### Migrations

We can create a server plugin to run the migrations in development automatically:

```ts [server/plugins/migrations.ts]
import { consola } from 'consola'
import { migrate } from 'drizzle-orm/d1/migrator'

export default defineNitroPlugin(async () => {
if (!import.meta.dev) return

onHubReady(async () => {
await migrate(useDrizzle(), { migrationsFolder: 'server/database/migrations' })
.then(() => {
consola.success('Database migrations done')
})
.catch((err) => {
consola.error('Database migrations failed', err)
})
})
})
```

::callout
Drizzle will create a `__drizzle_migrations` table in your database to keep track of the applied migrations. It will also run the migrations automatically in development mode.
::

To apply the migrations in staging or production, you can run the server using `npx nuxi dev --remote` command to connect your local server to the remote database, learn more about [remote storage](/docs/getting-started/remote-storage).
Migrations created with `npm run db:generate` are automatically applied during deployment, preview and when starting the development server.

::note
We are planning to update this section to leverage [Nitro Tasks](https://nitro.unjs.io/guide/tasks) instead of a server plugin in the future.
::note{to="/docs/features/database#migrations"}
Learn more about migrations.
::

### `useDrizzle()`
Expand Down
88 changes: 88 additions & 0 deletions docs/content/4.changelog/database-migrations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
---
title: Automatic Database Migrations
description: "Database migrations now automatically apply during development and deployment."
date: 2024-10-25
image: '/images/changelog/database-migrations.png'
category: Core
authors:
- name: Rihan Arfan
avatar:
src: https://avatars.githubusercontent.com/u/20425781?v=4
to: https://x.com/RihanArfan
username: RihanArfan
- name: Sebastien Chopin
avatar:
src: https://avatars.githubusercontent.com/u/904724?v=4
to: https://x.com/atinux
username: atinux
---

::tip
This feature is available on both [free and pro plans](/pricing) starting with [`@nuxthub/core >= v0.8.0`](https://github.com/nuxt-hub/core/releases).
::

We're excited to introduce automatic [database migrations](/docs/features/database#migrations) in NuxtHub.

### Automatic Migration Application

SQL migrations in `server/database/migrations/*.sql` now automatically apply when you:
- Start the development server (`npx nuxt dev` or [`npx nuxt dev --remote`](/docs/getting-started/remote-storage))
- Preview builds locally ([`npx nuxthub preview`](/changelog/nuxthub-preview))
- Deploy via [`npx nuxthub deploy`](/docs/getting-started/deploy#nuxthub-cli) or [Cloudflare Pages CI](/docs/getting-started/deploy#cloudflare-pages-ci)

Starting now, when you clone any of [our templates](/templates) with a database, all migrations apply automatically!

::note{to="/docs/features/database#migrations"}
Learn more about database migrations in our **full documentation**.
::

## New CLI Commands

[`[email protected]`](https://github.com/nuxt-hub/cli) introduces these database migration commands:

```bash [Terminal]
# Create a new migration
npx nuxthub database migrations create <name>

# View migration status
npx nuxthub database migrations list

# Mark all migrations as applied
npx nuxthub database migrations mark-all-applied
```

Learn more about:
- [Creating migrations](/docs/features/database#creating-migrations)
- [Checking migration status](/docs/features/database#checking-migration-status)
- [Marking migrations as applied](/docs/features/database#marking-migrations-as-applied)

## Migrating from Existing ORMs

::important
**Current Drizzle ORM users:** Follow these specific migration steps.
::

Since NuxtHub doesn't recognize previously applied Drizzle ORM migrations (stored in `__drizzle_migrations`), it will attempt to rerun all migrations in `server/database/migrations/*.sql`. To prevent this:

1. Mark existing migrations as applied in each environment:

```bash [Terminal]
# Local environment
npx nuxthub database migrations mark-all-applied

# Preview environment
npx nuxthub database migrations mark-all-applied --preview

# Production environment
npx nuxthub database migrations mark-all-applied --production
```

2. Remove `server/plugins/database.ts` as it's no longer needed.

## Understanding Database Migrations

Database migrations provide version control for your database schema. They track changes and ensure consistent schema evolution across all environments through incremental updates.

::note
Implemented in [nuxt-hub/core#333](https://github.com/nuxt-hub/core/pull/333) and [nuxt-hub/cli#31](https://github.com/nuxt-hub/cli/pull/31).
::
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 7 additions & 0 deletions playground/server/database/migrations/0001_create-todos.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
-- Migration number: 0001 2024-10-24T00:25:12.371Z
CREATE TABLE todos (
id integer PRIMARY KEY NOT NULL,
title text NOT NULL,
completed integer DEFAULT 0 NOT NULL,
created_at integer NOT NULL
);
14 changes: 0 additions & 14 deletions playground/server/plugins/migrations.ts

This file was deleted.

Loading