Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration fails #455

Open
svsk417 opened this issue Nov 18, 2024 · 0 comments
Open

Migration fails #455

svsk417 opened this issue Nov 18, 2024 · 0 comments

Comments

@svsk417
Copy link

svsk417 commented Nov 18, 2024

Hi!
Thanks for all the work on this project!

Currently I am doing the migration from an old version (0.88.13) to the newest one (0.111.8):
My first issues was, that the "legacy migrations" are not fully executed, because the relevant code just looks for the table to be legacy migration table to be existent, but not how many migrations already have been done. This is resulting in errors in migrations that are applied after those missing migrations.
See https://github.com/SigNoz/signoz-otel-collector/blob/v0.111.8/cmd/signozschemamigrator/schema_migrator/manager.go#L212
Relevant code for quick reference:

// shouldRunSquashed returns true if the legacy migrations table exists and the v2 table does not exist
func (m *MigrationManager) shouldRunSquashed(ctx context.Context, db string) (bool, error) {
	var count uint64
	err := m.conn.QueryRow(ctx, "SELECT count(*) FROM clusterAllReplicas($1, system.tables) WHERE database = $2 AND name = $3", m.clusterName, db, legacyMigrationsTable).Scan(&count)
	if err != nil {
		return false, err
	}
	var countV2 uint64
	err = m.conn.QueryRow(ctx, "SELECT count(*) FROM clusterAllReplicas($1, system.tables) WHERE database = $2 AND name = 'schema_migrations_v2'", m.clusterName, db).Scan(&countV2)
	if err != nil {
		return false, err
	}
	didMigrateV2 := false
	if countV2 > 0 {
		// if there are non-zero v2 migrations, we don't need to run the squashed migrations
		// fetch count of v2 migrations that are not finished
		var countMigrations uint64
		err = m.conn.QueryRow(ctx, fmt.Sprintf("SELECT count(*) FROM %s.distributed_schema_migrations_v2", db)).Scan(&countMigrations)
		if err != nil {
			return false, err
		}
		didMigrateV2 = countMigrations > 0
	}
	// we want to run the squashed migrations only if the legacy migrations do not exist
	// and v2 migrations did not run
	return count == 0 && !didMigrateV2, nil
}

Therefore, as a workaround I am first migrating to version 0.102.6, where the check did not exist and only afterwards to the newest one. Sadly this one also fails. When I enter the query that is failing with clickhouse-client, I get:

CREATE TABLE IF NOT EXISTS signoz_logs.logs_v2_resource UUID 'some-uuid'
(
    `labels` String CODEC(ZSTD(5)),
    `fingerprint` String CODEC(ZSTD(1)),
    `seen_at_ts_bucket_start` Int64 CODEC(Delta(8), ZSTD(1)),
    INDEX idx_labels lower(labels) TYPE ngrambf_v1(4, 1024, 3, 0) GRANULARITY 1
)
ENGINE = ReplicatedReplacingMergeTree
PARTITION BY toDate(seen_at_ts_bucket_start / 1000)
ORDER BY (labels, fingerprint, seen_at_ts_bucket_start)
TTL (toDateTime(seen_at_ts_bucket_start) + toIntervalSecond(1296000)) + toIntervalSecond(1800)
SETTINGS ttl_only_drop_parts = 1, index_granularity = 8192

Query id: some-uuid


Elapsed: 0.001 sec. 

Received exception from server (version 24.1.2):
Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: Macro 'uuid' and empty arguments of ReplicatedMergeTree are supported only for ON CLUSTER queries with Atomic database engine. (BAD_ARGUMENTS)

I must admit, that I am not a pro in Clickhouse. Therefore, any help is welcome. Even if it is just docs that I need to digest before I will be able to find the solutuon on my own.
Thanks in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant