Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blog: Update about namespace enforcement #6753

Merged
merged 3 commits into from
May 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions site/docs/blog/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
Highlights

* [Rolling upgrade issue to 0.26.0](incident-2022-05.md) (May 2022)
* [Namespace enforcement](namespace-enforcement.md) (May 2023)
13 changes: 13 additions & 0 deletions site/docs/blog/namespace-enforcement.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Namespace enforcement in Nessie

Starting from Nessie version 0.52.3, it is required to have existing Namespaces before creating or committing to tables.
In case of tables that have implicit namespaces, new commits will fail after the upgrade as this rule will be enforced.
Therefore, it is necessary to explicitly create the namespaces.

Namespaces can be created using any of these options:
* Using SQL command (for Spark, `CREATE NAMESPACE catalogName.namespaceName`)
* Using [Iceberg Catalog API](https://github.com/apache/iceberg/blob/3ab00171b48bb793a3b71845eb12d5077ba892f1/nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java#L262).
* Using [content generator tool](https://github.com/projectnessie/nessie/blob/main/tools/content-generator/README.md) to batch create the missing namespaces with `create-missing-namespaces` option.

[More info from the spec](https://github.com/projectnessie/nessie/blob/main/api/NESSIE-SPEC-2-0.md#200-beta1).
[Issue background](https://github.com/projectnessie/nessie/issues/6244).
44 changes: 37 additions & 7 deletions tools/content-generator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,43 @@ Functionality supported:
for "expected duration" divided by "number of commits" (default: no sleep)
* Generate content for N tables (default: 1 table)
* Generate content for any content type (default: Iceberg-table)
* Read references, commits and content objects
* Delete content objects
* Create missing namespaces


```
$ java -jar tools/content-generator/target/nessie-content-generator-0.12.2-SNAPSHOT.jar help generate
$ java -jar nessie-content-generator-0.58.1.jar -h
Usage: nessie-content-generator [-hV] [-u=<uri>] [COMMAND]
-h, --help Show this help message and exit.
-u, --uri=<uri> Nessie API endpoint URI, defaults to http://localhost:
19120/api/v2.
-V, --version Print version information and exit.
Commands:
generate Generate commits
commits Read commits
refs Read references
content Read content objects
content-refresh Get and Put content objects without changes to
refresh their storage model
delete Delete content objects
create-missing-namespaces Creates missing namespaces for content keys at
branch HEADs.
help Display help information about the specified
command.
```


Usage: nessie-content-generator generate [-hV] [-b=<branchCount>]
[-d=<runtimeDuration>] [-D=<defaultBranchName>] [-n=<numCommits>]
[-t=<numTables>] [-T=<newTagProbability>] [--type=<contentType>]
[-u=<uri>]
```
$ java -jar nessie-content-generator-0.58.1.jar help generate
Usage: nessie-content-generator generate [-hvV] [--continue-on-error]
[-b=<branchCount>] [-d=<runtimeDuration>] [-D=<defaultBranchName>]
[--key-pattern=<keyPattern>] [-n=<numCommits>]
[--puts-per-commit=<putsPerCommit>] [-t=<numTables>]
[-T=<newTagProbability>] [--type=<contentType>] [-u=<uri>]
Generate commits
-b, --num-branches=<branchCount>
Number of branches to use.
--continue-on-error
-d, --duration=<runtimeDuration>
Runtime duration, equally distributed among the
number of commits to create. See java.time.
Expand All @@ -30,8 +55,12 @@ Generate commits
Name of the default branch, uses the server's
default branch if not specified.
-h, --help Show this help message and exit.
--key-pattern=<keyPattern>
-n, --num-commits=<numCommits>
Number of commits to create.
--puts-per-commit=<putsPerCommit>
-t, --num-tables=<numTables>
Number of table names, each commit chooses a
random table (content-key).
Expand All @@ -42,6 +71,7 @@ Generate commits
ICEBERG_TABLE. Possible values: ICEBERG_TABLE,
ICEBERG_VIEW, DELTA_LAKE_TABLE
-u, --uri=<uri> Nessie API endpoint URI, defaults to http:
//localhost:19120/api/v1.
//localhost:19120/api/v2.
-v, --verbose Produce verbose output (if possible)
-V, --version Print version information and exit.
```