Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore backup created by 0.6.4 with clickhouse-backup 1.0.0 #268

Closed
rodrigargar opened this issue Sep 15, 2021 · 0 comments · Fixed by #288
Closed

Restore backup created by 0.6.4 with clickhouse-backup 1.0.0 #268

rodrigargar opened this issue Sep 15, 2021 · 0 comments · Fixed by #288
Assignees

Comments

@rodrigargar
Copy link
Contributor

When restoring with clickhouse-backup 1.0.0 a backup created with a previous version of clickhouse-backup (0.6.4), I found a couple of issues:

  1. Downloading the backup gives an is empty backup error, although the backup is not empty:

    $ clickhouse-backup download 2021-09-15-rnd-0.6.4-shard02
    2021/09/15 16:06:49 error '2021-09-15-rnd-0.6.4-shard02' is empty backup

@Slach gave me a workaround for this: allow empty backups in the configuration of clickhouse-backup.

general:
  allow_empty_backups: true
  1. Restoring the schema tries to create the tables on the system database instead of the appropriate database.

In my case, the database rnd holds the events_local table and the events distributed table. When restoring the schema, clickhouse-backup tries to create the tables system.events_local and system.events (which already exists):

# LOG_LEVEL=debug clickhouse-backup restore --rm --schema 2021-09-15-rnd-0.6.4-shard02
...
2021/09/15 16:17:02 debug CREATE DATABASE IF NOT EXISTS `rnd`
2021/09/15 16:17:02 debug SELECT engine FROM system.databases WHERE name = 'rnd'
2021/09/15 16:17:02 debug DROP TABLE IF EXISTS `rnd`.`events_local`
2021/09/15 16:17:02 debug CREATE TABLE events_local
(
    `time` DateTime,
    `kind` UInt8,
    `host` String,
    `port` UInt16,
    `data` UInt32
)
ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{shard}/rnd.events_local', '{shard}_{replica}')
PARTITION BY toYear(time)
ORDER BY (time, kind)
SETTINGS index_granularity = 8192

2021/09/15 16:17:03 debug CREATE DATABASE IF NOT EXISTS `rnd`
2021/09/15 16:17:03 debug SELECT engine FROM system.databases WHERE name = 'rnd'
2021/09/15 16:17:03 debug DROP TABLE IF EXISTS `rnd`.`events`
2021/09/15 16:17:03 debug CREATE TABLE events
(
    `time` DateTime,
    `kind` UInt8,
    `host` String,
    `port` UInt16,
    `data` UInt32
)
ENGINE = Distributed('{cluster}', 'rnd', 'events_local', rand())

2021/09/15 16:17:03  warn can't create table 'rnd.events': code: 57, message: Table system.events already exists., will try again

Table events_local table is indeed created in the system database instead of the rnd database:

SHOW CREATE TABLE system.events_local

Row 1:
──────
statement: CREATE TABLE system.events_local
(
    `time` DateTime,
    `kind` UInt8,
    `host` String,
    `port` UInt16,
    `data` UInt32
)
ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{shard}/rnd.events_local', '{shard}_{replica}')
PARTITION BY toYear(time)
ORDER BY (time, kind)
SETTINGS index_granularity = 8192

Clickhouse-backup versions:

To create the backup:

$ clickhouse-backup --version
Version:         0.6.4
Git Commit:      4a0b3c040783232066f5b7ca4b2990d6ef50db19
Build Date:      2021-01-27

To restore the backup:

$ clickhouse-backup --version
Version:         1.0.0
Git Commit:      37f3bd78adec2aadc1de10f9323fe426a5e12dc4
Build Date:      2021-06-16
@Slach Slach self-assigned this Sep 16, 2021
@Slach Slach mentioned this issue Oct 24, 2021
Slach added a commit that referenced this issue Oct 24, 2021
# v1.2.0

INCOMPATIBLE CHANGES
- REST API `/backup/status` now return only latest executed command with status and error message

IMPROVEMENTS
- Added REST API `/backup/list/local` and `/backup/list/remote` to allow list backup types separately
- Decreased background backup creation time via REST API `/backup/create`, during avoid list remote backups for update metrics value 
- Decreased backup creation time, during avoid scan whole `system.tables` when set `table` query string parameter or `--tables` cli parameter     
- Added `last` and `filter` query string parameters to REST API `/backup/actions`, to avoid pass to client long JSON documents
- Improved `FTP` remote storage parallel upload / download
- Added `FTP_CONCURRENCY` to allow, by default MAX_CPU / 2 
- Added `FTP_DEBUG` setting, to allow debug FTP commands
- Added `FTP` to CI/CD on any commit
- Added race condition check to CI/CD

BUG FIXES
- environment variable `LOG_LEVEL` now apply to `clickhouse-backup server` properly
- fix #280, incorrect prometheus metrics measurement for `/backup/create`, `/backup/upload`, `/backup/download`
- fix #273, return `S3_PART_SIZE` back, but calculates it smartly
- fix #252, now you can pass `last` and `filter` query string parameters
- fix #246, incorrect error messages when use `REMOTE_STORAGE=none`
- fix #283, properly handle error message from `FTP` server 
- fix #268, properly restore legacy backup for schema without database name
- fix #287
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants